Oct 30 21:45:35.645893 ip-10-0-137-17 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:45:36.107500 ip-10-0-137-17 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:45:36.107500 ip-10-0-137-17 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:45:36.107500 ip-10-0-137-17 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:45:36.108035 ip-10-0-137-17 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:45:36.112129 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112049 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.112129 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112125 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.112129 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112130 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.112129 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112134 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112138 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112142 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112146 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112150 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112154 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112157 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112162 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112167 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112171 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112175 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112179 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112183 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112186 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112190 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112194 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112199 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112203 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112207 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.112358 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112211 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112215 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112219 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112223 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112229 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112242 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112249 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112253 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112258 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112262 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112267 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112271 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112275 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112279 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112284 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112289 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112294 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112299 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112303 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.113141 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112308 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112312 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112316 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112321 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112325 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112330 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112334 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112338 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112343 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112347 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112351 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112355 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112360 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112366 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112370 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112375 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112379 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112383 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112387 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112392 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.113931 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112396 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112400 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112404 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112408 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112413 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112417 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112421 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112425 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112430 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112435 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112439 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112444 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112448 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112452 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112456 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112460 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112465 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112469 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112474 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112478 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.114787 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112483 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112487 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112493 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112497 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112502 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112506 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112510 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112515 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112519 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112524 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112528 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112532 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112536 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112540 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112545 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112549 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112553 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112557 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112561 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.115624 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112566 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112575 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112581 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112586 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112591 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.112595 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113887 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113897 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113903 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113907 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113911 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113915 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113920 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113924 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113928 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113932 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113941 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113946 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113950 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113954 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.116455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113958 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113962 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113967 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113971 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113976 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113980 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113985 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113989 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113993 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.113997 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114001 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114006 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114012 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114016 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114020 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114025 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114029 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114033 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114037 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114042 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.117386 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114046 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114050 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114054 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114058 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114063 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114067 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114071 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114075 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114079 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114085 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114089 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114093 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114097 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114102 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114106 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114110 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114114 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114118 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114123 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.118219 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114128 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114132 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114136 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114140 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114144 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114149 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114154 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114158 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114162 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114166 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114172 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114176 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114180 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114185 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114189 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114193 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114197 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114201 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114206 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.118806 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114210 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114214 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114219 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114223 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114228 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114232 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114236 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114240 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114244 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114249 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114253 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114257 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114261 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114266 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114270 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114274 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114278 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114282 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114287 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.119369 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114292 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114296 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114300 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114304 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114311 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114318 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114322 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114327 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114331 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114335 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114343 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114349 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114354 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114358 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.114362 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115185 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115193 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115198 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115203 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.119850 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115207 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115212 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115216 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115220 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115225 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115229 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115234 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115238 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115242 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115246 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115250 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115255 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115259 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115264 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115268 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115272 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115277 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115281 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115285 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115290 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.120387 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115294 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115298 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115302 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115307 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115311 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115316 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115320 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115324 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115328 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115333 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115340 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115347 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115352 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115357 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115362 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115366 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115370 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115375 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.120888 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115379 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115383 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115387 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115392 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115404 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115408 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115412 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115416 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115420 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115425 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115429 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115433 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115438 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115442 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115446 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115450 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115454 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115458 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115462 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.121316 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115466 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115471 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115475 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115479 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115483 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115487 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115491 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115496 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115500 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115504 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115509 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115513 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115517 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115521 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115527 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115534 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115539 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115543 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115547 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.121851 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115551 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115556 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115560 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115564 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115568 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115575 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115579 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115585 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115590 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115594 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115598 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115602 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115606 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115610 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115615 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115619 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115643 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115648 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115652 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115656 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.122298 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115660 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115666 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115670 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115674 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115678 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.115682 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115811 2571 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115822 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115832 2571 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115839 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115846 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115852 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115858 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115864 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115870 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115874 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115880 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115887 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115892 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115899 2571 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115903 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115908 2571 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115913 2571 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:45:36.122813 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115918 2571 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115925 2571 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115929 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115934 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115939 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115945 2571 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115951 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115955 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115961 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115966 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115971 2571 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115976 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115981 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115986 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115991 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.115997 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116002 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116007 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116012 2571 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116017 2571 flags.go:64] FLAG: --enable-server="true" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116021 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116029 2571 flags.go:64] FLAG: --event-burst="100" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116034 2571 flags.go:64] FLAG: --event-qps="50" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116038 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:45:36.123399 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116043 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116047 2571 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116056 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116060 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116067 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116072 2571 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116076 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116081 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116085 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116090 2571 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116095 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116100 2571 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116104 2571 flags.go:64] FLAG: --feature-gates="" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116114 2571 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116120 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116125 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116130 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116135 2571 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116140 2571 flags.go:64] FLAG: --help="false" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116145 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116151 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116156 2571 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116161 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:45:36.124058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116166 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116172 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116177 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116181 2571 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116186 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116191 2571 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116195 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116201 2571 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116205 2571 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116209 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116214 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116221 2571 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116226 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116232 2571 flags.go:64] FLAG: --lock-file="" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116237 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116241 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116246 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116255 2571 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116260 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116265 2571 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116270 2571 flags.go:64] FLAG: --logging-format="text" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116275 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116280 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116285 2571 flags.go:64] FLAG: --manifest-url="" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116290 2571 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:45:36.124600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116296 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116301 2571 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116308 2571 flags.go:64] FLAG: --max-pods="110" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116313 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116318 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116323 2571 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116328 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116333 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116338 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116342 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116353 2571 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116358 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116363 2571 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116367 2571 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116372 2571 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116376 2571 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116381 2571 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116386 2571 flags.go:64] FLAG: --port="10250" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116393 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116398 2571 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01c05a4f2dc47ba5a" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116405 2571 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116410 2571 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116415 2571 flags.go:64] FLAG: --register-node="true" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116419 2571 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116426 2571 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116431 2571 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:45:36.125212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116435 2571 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116440 2571 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116446 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116451 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116455 2571 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116460 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116465 2571 flags.go:64] FLAG: --runonce="false" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116469 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116473 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116478 2571 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116483 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116488 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116498 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116503 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116507 2571 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116512 2571 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116516 2571 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116521 2571 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116526 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116531 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116536 2571 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116540 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116549 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116553 2571 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116559 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:45:36.125840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116565 2571 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116573 2571 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116577 2571 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116582 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116587 2571 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116591 2571 flags.go:64] FLAG: --v="2" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116598 2571 flags.go:64] FLAG: --version="false" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116605 2571 flags.go:64] FLAG: --vmodule="" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116611 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.116616 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116865 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116873 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116879 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116883 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116887 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116892 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116896 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116902 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116908 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116912 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116917 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.126457 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116922 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116926 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116930 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116935 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116939 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116943 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116947 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116952 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116956 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116960 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116967 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116973 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116981 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116985 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116991 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.116996 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117001 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117005 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.126968 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117009 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117014 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117018 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117022 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117026 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117030 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117034 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117038 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117043 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117047 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117051 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117055 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117060 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117065 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117070 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117075 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117079 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117083 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117088 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117092 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.127401 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117096 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117101 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117105 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117109 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117115 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117120 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117126 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117130 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117135 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117139 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117143 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117148 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117152 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117156 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117160 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117164 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117169 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117173 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117178 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117183 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.127922 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117188 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117192 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117197 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117201 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117205 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117210 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117214 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117218 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117223 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117227 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117231 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117235 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117240 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117244 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117248 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117253 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117259 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117263 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117270 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.128392 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117292 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117297 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117303 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117307 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117312 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117316 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117321 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117325 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117329 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117333 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117337 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117342 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117345 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117350 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117354 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117359 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117363 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.128858 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117367 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.117406 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117465 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117474 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117481 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117486 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117490 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117494 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117498 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117503 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117507 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117511 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117515 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117521 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117527 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.129264 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117531 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117535 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117540 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117544 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117548 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117552 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117557 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117561 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117565 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117569 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117573 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117577 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117581 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117586 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117590 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117594 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117598 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117603 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117607 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117611 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.129618 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117615 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117619 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117640 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117646 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117650 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117654 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117659 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117663 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117667 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117671 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117698 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117707 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117713 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117717 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117722 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117726 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117732 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117736 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117741 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117745 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.130102 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117749 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117754 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117758 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117762 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117766 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117770 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117775 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117780 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117784 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117795 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117799 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117804 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117808 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117812 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117816 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117821 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117826 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117830 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117834 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117839 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.130563 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117843 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117847 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117853 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117860 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117864 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117868 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117873 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117877 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117881 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117886 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117890 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117894 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117899 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117903 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117907 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117912 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117916 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117920 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117928 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117934 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.131040 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117940 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117944 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117949 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117953 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117958 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117962 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117967 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117971 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117976 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117980 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117984 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117988 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.117992 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.124602 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.124616 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.124643 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.124657 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124710 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124716 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.131500 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124719 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124722 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124725 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124728 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124730 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124733 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124736 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124739 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124742 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124744 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124747 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124750 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124752 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124755 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124758 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124760 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124763 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124766 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124768 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.131998 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124771 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124774 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124777 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124780 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124782 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124785 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124787 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124790 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124793 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124796 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124799 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124801 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124804 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124806 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124809 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124812 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124815 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124817 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124820 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124822 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124825 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.132441 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124827 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124830 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124833 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124835 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124838 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124841 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124843 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124846 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124848 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124851 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124854 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124856 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124859 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124861 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124865 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124868 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124871 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124873 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124876 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.132955 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124879 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124882 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124884 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124887 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124890 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124892 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124895 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124898 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124900 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124903 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124905 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124908 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124910 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124913 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124915 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124918 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124920 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124924 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124928 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.133399 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124931 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124934 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124936 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124939 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124942 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124944 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124947 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124949 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124952 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124956 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124958 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124961 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124963 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124967 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124969 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124972 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124974 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124977 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124979 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.134117 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124982 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124985 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124987 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124990 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124992 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124994 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.124998 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.125026 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125165 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125170 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125174 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125177 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125180 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125182 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125185 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.134649 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125189 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125193 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125195 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125198 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125201 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125204 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125206 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125209 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125212 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125214 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125217 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125220 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125223 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125225 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125228 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125231 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125233 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125236 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125238 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125241 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.135004 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125244 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125246 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125249 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125251 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125253 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125256 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125259 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125261 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125263 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125266 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125268 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125271 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125274 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125276 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125279 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125281 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125284 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125286 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125289 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.135506 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125291 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125294 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125297 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125300 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125303 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125305 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125308 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125310 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125313 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125315 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125318 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125320 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125323 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125325 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125328 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125330 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125333 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125335 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125338 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.136083 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125340 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125343 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125345 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125348 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125350 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125353 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125355 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125358 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125360 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125363 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125365 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125369 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125372 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125375 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125378 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125380 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125383 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125386 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125389 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.136527 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125392 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125395 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125397 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125400 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125402 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125405 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125407 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125410 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125412 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125415 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125417 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125420 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125422 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125425 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125427 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125429 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125432 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125435 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125437 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125440 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.136993 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125442 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125444 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.125467 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125498 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125503 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125506 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125510 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125513 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125516 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125519 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125522 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125525 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125528 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125531 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125533 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125536 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:36.137458 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125538 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125541 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125543 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125546 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125549 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125552 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125554 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125557 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125559 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125562 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125564 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125567 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125570 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125572 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125574 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125577 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125579 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125582 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125585 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125587 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:36.137861 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125590 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125592 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125595 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125597 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125600 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125603 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125605 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125608 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125611 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125613 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125616 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125619 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125621 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125624 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125637 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125640 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125643 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125645 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125648 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:36.138320 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125650 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125653 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125655 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125658 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125660 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125663 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125665 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125668 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125670 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125673 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125676 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125678 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125681 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125683 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125686 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125688 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125691 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125693 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125697 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:36.138860 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125701 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125704 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125707 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125710 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125713 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125715 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125720 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125722 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125725 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125727 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125730 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125732 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125735 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125737 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125740 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125742 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125745 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125747 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125750 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125752 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:36.139294 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125755 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125757 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125760 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125762 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125765 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125767 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125770 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125772 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125775 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125778 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125780 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125784 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125793 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125797 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:36.125802 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.125851 2571 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.127925 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.129029 2571 server.go:1014] "Starting client certificate rotation" Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.129127 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:45:36.139767 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.130011 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:45:36.140243 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.131109 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:45:36.140243 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.132192 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:45:36.147169 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.147154 2571 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:45:36.151027 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.151013 2571 log.go:25] "Validated CRI v1 image API" Oct 30 21:45:36.151840 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.151811 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:45:36.155411 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.155393 2571 fs.go:135] Filesystem UUIDs: map[22729a8c-cada-4251-8364-537976f5e64c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d587f00f-3c55-4d3b-9e2c-d92e16d0704f:/dev/nvme0n1p4] Oct 30 21:45:36.155468 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.155411 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:45:36.160979 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.160877 2571 manager.go:217] Machine: {Timestamp:2026-10-30 21:45:36.159309852 +0000 UTC m=+0.383604885 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3129090 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2958a22a3aae2ccad3d305d5a952d0 SystemUUID:ec2958a2-2a3a-ae2c-cad3-d305d5a952d0 BootID:7977c813-1676-484b-8ad6-63067469be66 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f7:05:6d:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f7:05:6d:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:c8:40:19:ae:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:45:36.160979 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.160979 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:45:36.161090 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.161077 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:45:36.165261 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.165242 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:45:36.165383 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.165262 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-17.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:45:36.165430 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.165385 2571 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:45:36.165430 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.165394 2571 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:45:36.165430 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.165418 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:45:36.165430 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.165430 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:45:36.166318 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.166307 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:45:36.170071 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.170060 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:45:36.170173 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.170164 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:45:36.170918 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.170909 2571 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:45:36.170951 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.170922 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:45:36.170951 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.170933 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:45:36.170951 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.170941 2571 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:45:36.170951 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.170949 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:45:36.171212 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.171199 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:45:36.174096 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.174073 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:45:36.181005 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.180988 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:45:36.181799 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181780 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:45:36.181842 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181817 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:45:36.181962 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181952 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:45:36.182004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181967 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:45:36.182004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181973 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:45:36.182004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181979 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:45:36.182004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181985 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:45:36.182004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.181990 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:45:36.182790 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.182781 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:45:36.182820 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.182791 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:45:36.182820 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.182798 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:45:36.182820 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.182805 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:45:36.182820 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.182813 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:45:36.182977 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.182822 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:45:36.184302 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.184292 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:45:36.184302 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.184301 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:45:36.187868 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.187857 2571 server.go:1263] "Started kubelet" Oct 30 21:45:36.188010 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.187957 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:45:36.188055 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.187969 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:45:36.188055 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.188052 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:45:36.188501 ip-10-0-137-17 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:45:36.189425 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.189402 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:45:36.190555 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.190533 2571 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:45:36.195392 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.195374 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:45:36.195864 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.195846 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:45:36.196002 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.195972 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:36.196071 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.196039 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:45:36.196641 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.196605 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:45:36.196641 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.196620 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:45:36.196774 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.196652 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:45:36.196774 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.196748 2571 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:45:36.196774 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.196755 2571 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:45:36.196890 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.196808 2571 factory.go:55] Registering systemd factory Oct 30 21:45:36.196890 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.196837 2571 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:45:36.197131 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.197117 2571 factory.go:153] Registering CRI-O factory Oct 30 21:45:36.197195 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.197150 2571 factory.go:223] Registration of the crio container factory successfully Oct 30 21:45:36.197262 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.197250 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:45:36.197297 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.197272 2571 factory.go:103] Registering Raw factory Oct 30 21:45:36.197297 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.197287 2571 manager.go:1199] Started watching for new ooms in manager Oct 30 21:45:36.198101 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.198082 2571 manager.go:319] Starting recovery of all containers Oct 30 21:45:36.203463 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.203441 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:45:36.208353 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.208335 2571 manager.go:324] Recovery completed Oct 30 21:45:36.214909 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.214890 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.216345 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.216326 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.216421 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.216356 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.216421 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.216368 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.216952 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.216939 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.217035 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.216939 2571 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:45:36.217035 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.216974 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:45:36.217035 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.216991 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:45:36.217742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.217724 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.217824 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.217747 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.217824 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.217756 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.219541 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.219529 2571 policy_none.go:50] "Start" Oct 30 21:45:36.219582 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.219544 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:45:36.219582 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.219559 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:45:36.221601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.221589 2571 policy_none.go:44] "Start" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.248462 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.251043 2571 manager.go:341] "Starting Device Plugin manager" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.251545 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.251559 2571 server.go:87] "Starting device plugin registration server" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.251753 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.251763 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.251848 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.251930 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.251938 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.252393 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.252432 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.260461 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.260478 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.260494 2571 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:45:36.273335 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.260525 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:45:36.325054 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.325035 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-17.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:45:36.325054 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.325033 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-17.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:45:36.326175 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.325036 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-17.us-west-2.compute.internal.18e36cd469d217ba default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-17.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-17.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:36.18783225 +0000 UTC m=+0.412127281,LastTimestamp:2026-10-30 21:45:36.18783225 +0000 UTC m=+0.412127281,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-17.us-west-2.compute.internal,}" Oct 30 21:45:36.352679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.352661 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.353364 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.353345 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.353443 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.353373 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.353443 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.353389 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.353443 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.353415 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.361234 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.361193 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal"] Oct 30 21:45:36.361301 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.361242 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.361933 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.361912 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.362008 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.361943 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.362008 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.361954 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.364051 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364039 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.364198 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364181 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.364248 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364214 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.364757 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364742 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.364838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364757 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.364838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364783 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.364838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364784 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.364838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364794 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.364838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.364799 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.366856 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.366841 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.366939 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.366867 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.367500 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.367484 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.367584 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.367520 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.367584 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.367535 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.401536 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.401521 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.404367 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.404353 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.404797 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.404783 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:45:36.404868 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.404842 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vmxq" Oct 30 21:45:36.410672 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.410653 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:45:36.410958 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.410943 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-17.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:45:36.426292 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.426264 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vmxq" Oct 30 21:45:36.457245 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.457176 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-17.us-west-2.compute.internal.18e36cd46b8523df default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-17.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-17.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-17.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:36.216343519 +0000 UTC m=+0.440638552,LastTimestamp:2026-10-30 21:45:36.216343519 +0000 UTC m=+0.440638552,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-17.us-west-2.compute.internal,}" Oct 30 21:45:36.481776 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.481755 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:36.481867 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.481853 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:36.482441 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.482427 2571 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-17.us-west-2.compute.internal\" not found" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.498173 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.498154 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.498228 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.498180 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.498228 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.498200 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44d4a0aec485b9ad44110c51253a50d0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"44d4a0aec485b9ad44110c51253a50d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.598493 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.598472 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44d4a0aec485b9ad44110c51253a50d0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"44d4a0aec485b9ad44110c51253a50d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.598577 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.598500 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.598577 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.598518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.598577 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.598542 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.598577 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.598564 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44d4a0aec485b9ad44110c51253a50d0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"44d4a0aec485b9ad44110c51253a50d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.598723 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.598575 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.624834 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.624799 2571 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-17.us-west-2.compute.internal\" not found" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.683326 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.683311 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:36.684177 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.684162 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:36.684271 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.684195 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:36.684271 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.684211 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:36.684271 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.684243 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.704734 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.704716 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.709190 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.708752 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.750965 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:36.750948 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:36.751032 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.750968 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-17.us-west-2.compute.internal\": node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:36.863711 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.863693 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:36.964050 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:36.964031 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:38.325094 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:38.325063 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:38.389242 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.389221 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:45:38.389655 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.389392 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:45:38.389655 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.389391 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:45:38.389655 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:38.389368 2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a26314a26c7f3463b9bcb9cfe0458715-c9cd20b13308b9a1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-17.us-west-2.compute.internal.18e36cd46b85677d\": read tcp 10.0.137.17:32866->3.151.40.68:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-17.us-west-2.compute.internal.18e36cd46b85677d default 4781 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-17.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-17.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-17.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:36 +0000 UTC,LastTimestamp:2026-10-30 21:45:36.36194805 +0000 UTC m=+0.586243081,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-17.us-west-2.compute.internal,}" Oct 30 21:45:38.425507 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:38.425479 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:38.526286 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:38.526102 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:38.627259 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:38.627224 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:38.691593 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.691544 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:40:37 +0000 UTC" deadline="2028-03-27 02:44:37.052603126 +0000 UTC" Oct 30 21:45:38.691593 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.691583 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12316h58m58.361025322s" Oct 30 21:45:38.727802 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:38.727767 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:45:38.760279 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.760253 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:38.857529 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.857461 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:38.916185 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.916163 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:45:38.918400 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.918388 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:45:38.978385 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:38.978363 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:45:39.130455 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.130426 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf14b302ecfd0d0d1206ce2297bec21.slice/crio-a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a WatchSource:0}: Error finding container a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a: Status 404 returned error can't find the container with id a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a Oct 30 21:45:39.130681 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.130661 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d4a0aec485b9ad44110c51253a50d0.slice/crio-05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924 WatchSource:0}: Error finding container 05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924: Status 404 returned error can't find the container with id 05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924 Oct 30 21:45:39.135619 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.135605 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:45:39.252225 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.252205 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:39.348103 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.348083 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:39.401734 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.401687 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:39.432849 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.432830 2571 apiserver.go:52] "Watching apiserver" Oct 30 21:45:39.457314 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.457284 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:45:39.525747 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.525709 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"44d4a0aec485b9ad44110c51253a50d0","Type":"ContainerStarted","Data":"05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924"} Oct 30 21:45:39.526801 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.526783 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"fdf14b302ecfd0d0d1206ce2297bec21","Type":"ContainerStarted","Data":"a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a"} Oct 30 21:45:39.530565 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.530551 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:45:39.530818 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.530804 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-r2dj5","kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal","openshift-image-registry/node-ca-xm8l9","openshift-multus/multus-746ws","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-wbmj9","openshift-network-operator/iptables-alerter-fmwmz","kube-system/global-pull-secret-syncer-4zhvv","openshift-dns/node-resolver-g66br","openshift-network-diagnostics/network-check-target-tqcc7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg","openshift-cluster-node-tuning-operator/tuned-vwfkb","openshift-multus/multus-additional-cni-plugins-vznvs","openshift-ovn-kubernetes/ovnkube-node-lgx8f"] Oct 30 21:45:39.534562 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.534548 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.536597 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.536579 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.538709 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.538693 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-746ws" Oct 30 21:45:39.540789 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.540771 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:39.540878 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.540829 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:39.542808 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.542791 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.544919 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.544904 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.545041 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.544952 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:39.547600 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.547582 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.550117 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.550096 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:39.550198 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.550144 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:39.552334 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.552306 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.554646 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.554629 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.556882 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.556864 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.559104 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.559086 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.568621 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568603 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-slash\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.568700 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-konnectivity-ca\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.568700 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-multus-certs\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.568700 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568658 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:39.568700 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-conf\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.568838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568733 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntnrc\" (UniqueName: \"kubernetes.io/projected/2a5249fa-084f-451b-8e94-9cbd497a8ea7-kube-api-access-ntnrc\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.568838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568771 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16867cf6-0526-47b2-866e-5dfc13f04869-tmp-dir\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.568838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568796 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-hostroot\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.568919 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysconfig\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.568919 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568877 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-tmp\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.568919 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cnibin\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.569004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.569004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568954 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.569004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568975 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.569004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.568998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-device-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.569122 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569013 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-log-socket\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569122 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569037 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569122 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569063 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-cni-binary-copy\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.569122 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-kubelet\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569236 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569134 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-ovn\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569236 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569167 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-socket-dir-parent\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.569236 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569190 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552b3503-9502-4ac7-af98-0acbc947254c-host-slash\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.569236 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569231 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.569381 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-kubernetes\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.569381 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569269 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-systemd\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.569381 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-var-lib-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569381 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569310 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-bin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.569544 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569421 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:39.569544 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-modprobe-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.569544 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569471 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-env-overrides\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569544 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569498 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-agent-certs\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.569544 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569525 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-cnibin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.569679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/552b3503-9502-4ac7-af98-0acbc947254c-iptables-alerter-script\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.569679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569570 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-dbus\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.569679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569602 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47qr\" (UniqueName: \"kubernetes.io/projected/12b9b148-13a4-40f3-9113-2877f5ca9574-kube-api-access-j47qr\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.569679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-bin\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-sys\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.569679 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-system-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569747 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-etc-tuned\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569777 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-registration-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569807 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-node-log\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22qrq\" (UniqueName: \"kubernetes.io/projected/dbdaddf0-875f-46f2-8418-62377b63a086-kube-api-access-22qrq\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-host\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569901 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-run\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.569957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569977 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-sys-fs\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.569992 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-systemd-units\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570043 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-netns\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570066 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jtl\" (UniqueName: \"kubernetes.io/projected/50406e5f-7f63-49e5-aca1-d572b155292a-kube-api-access-f5jtl\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570128 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-config\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570167 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-kubelet-config\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570199 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b9b148-13a4-40f3-9113-2877f5ca9574-host\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-netns\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-systemd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkccl\" (UniqueName: \"kubernetes.io/projected/f008c236-2573-46f0-bbdd-f10daa5d08bf-kube-api-access-lkccl\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570305 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-kubelet\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4qt\" (UniqueName: \"kubernetes.io/projected/5af23954-d911-4664-872d-9ae2e67ecfa0-kube-api-access-cw4qt\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58vn\" (UniqueName: \"kubernetes.io/projected/16867cf6-0526-47b2-866e-5dfc13f04869-kube-api-access-l58vn\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570431 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570455 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-etc-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570521 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-k8s-cni-cncf-io\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-daemon-config\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570553 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-netd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-conf-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-etc-kubernetes\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570642 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82np\" (UniqueName: \"kubernetes.io/projected/552b3503-9502-4ac7-af98-0acbc947254c-kube-api-access-d82np\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-var-lib-kubelet\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570685 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b9b148-13a4-40f3-9113-2877f5ca9574-serviceca\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovn-node-metrics-cert\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.570753 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-script-lib\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.571203 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570740 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-multus\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.571203 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570753 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-lib-modules\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.571203 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16867cf6-0526-47b2-866e-5dfc13f04869-hosts-file\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.571203 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570813 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-socket-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.571203 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.570845 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtl9\" (UniqueName: \"kubernetes.io/projected/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kube-api-access-vjtl9\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.584526 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.584513 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:45:39.584592 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.584542 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kjn6x\"" Oct 30 21:45:39.584592 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.584580 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:45:39.629819 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.629794 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:45:39.629906 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.629844 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cfzkv" Oct 30 21:45:39.658080 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.658038 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:45:39.671909 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.671895 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b9b148-13a4-40f3-9113-2877f5ca9574-host\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.671972 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.671916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-netns\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.671972 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.671930 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-systemd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.671972 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.671944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkccl\" (UniqueName: \"kubernetes.io/projected/f008c236-2573-46f0-bbdd-f10daa5d08bf-kube-api-access-lkccl\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.671972 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.671957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-kubelet\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672002 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-netns\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.671999 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4qt\" (UniqueName: \"kubernetes.io/projected/5af23954-d911-4664-872d-9ae2e67ecfa0-kube-api-access-cw4qt\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672022 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b9b148-13a4-40f3-9113-2877f5ca9574-host\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58vn\" (UniqueName: \"kubernetes.io/projected/16867cf6-0526-47b2-866e-5dfc13f04869-kube-api-access-l58vn\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-kubelet\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672067 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672029 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-systemd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672094 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-etc-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672119 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-k8s-cni-cncf-io\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-daemon-config\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672172 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672199 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-etc-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672199 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-netd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672209 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-k8s-cni-cncf-io\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-conf-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672238 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-netd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672258 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-etc-kubernetes\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672274 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82np\" (UniqueName: \"kubernetes.io/projected/552b3503-9502-4ac7-af98-0acbc947254c-kube-api-access-d82np\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672288 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-conf-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672309 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-etc-kubernetes\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672313 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-var-lib-kubelet\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b9b148-13a4-40f3-9113-2877f5ca9574-serviceca\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672379 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-var-lib-kubelet\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672386 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovn-node-metrics-cert\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672410 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-script-lib\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.672601 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672495 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-multus\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672566 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-multus\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-lib-modules\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16867cf6-0526-47b2-866e-5dfc13f04869-hosts-file\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672818 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-socket-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtl9\" (UniqueName: \"kubernetes.io/projected/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kube-api-access-vjtl9\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-slash\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-konnectivity-ca\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.672998 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-lib-modules\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673131 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-socket-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.673299 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673200 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16867cf6-0526-47b2-866e-5dfc13f04869-hosts-file\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.673833 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673474 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-konnectivity-ca\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.673833 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673556 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-slash\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.673833 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-multus-certs\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.673833 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673699 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:39.673833 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673743 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-conf\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.673833 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntnrc\" (UniqueName: \"kubernetes.io/projected/2a5249fa-084f-451b-8e94-9cbd497a8ea7-kube-api-access-ntnrc\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.673833 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673814 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16867cf6-0526-47b2-866e-5dfc13f04869-tmp-dir\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.674136 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673845 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-hostroot\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.674136 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysconfig\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.674136 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.673914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-tmp\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.674136 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674045 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-conf\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.674136 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674094 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cnibin\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.674136 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674134 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.674423 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.674423 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.674423 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674236 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-device-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.674423 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674270 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-log-socket\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.674423 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674306 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.674423 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674279 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:45:39.674687 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674526 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-multus-certs\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.674687 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.674639 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:39.674782 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.674714 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:40.174688185 +0000 UTC m=+3.138292617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:39.674782 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674761 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cnibin\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.674958 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.674938 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.675091 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675054 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-device-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.675146 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675106 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-log-socket\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.675188 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675158 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.675229 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675205 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-hostroot\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.675487 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675469 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16867cf6-0526-47b2-866e-5dfc13f04869-tmp-dir\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.675648 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675629 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-cni-binary-copy\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.675733 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675663 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-kubelet\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.675733 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675693 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-ovn\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.675733 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-socket-dir-parent\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.675893 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675735 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysconfig\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.675893 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552b3503-9502-4ac7-af98-0acbc947254c-host-slash\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.675893 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.675893 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.675879 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675905 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-kubernetes\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675973 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-socket-dir-parent\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675982 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-ovn\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675974 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552b3503-9502-4ac7-af98-0acbc947254c-host-slash\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.675970 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret podName:d3280f18-2d95-4303-8f34-5a2fe91671e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:40.175922965 +0000 UTC m=+3.139527390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret") pod "global-pull-secret-syncer-4zhvv" (UID: "d3280f18-2d95-4303-8f34-5a2fe91671e6") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.675882 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-kubelet\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-kubernetes\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-systemd\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-var-lib-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-var-lib-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-bin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676447 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-systemd\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676485 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-modprobe-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676498 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-bin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.676513 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-env-overrides\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-agent-certs\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-cnibin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/552b3503-9502-4ac7-af98-0acbc947254c-iptables-alerter-script\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676586 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-modprobe-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-dbus\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676623 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47qr\" (UniqueName: \"kubernetes.io/projected/12b9b148-13a4-40f3-9113-2877f5ca9574-kube-api-access-j47qr\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676646 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676650 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-cnibin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-bin\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-sys\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676696 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-dbus\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676716 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676741 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-bin\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676742 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-system-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676752 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-etc-tuned\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676783 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-sys\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.676879 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676796 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-registration-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676820 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-node-log\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676827 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-system-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qrq\" (UniqueName: \"kubernetes.io/projected/dbdaddf0-875f-46f2-8418-62377b63a086-kube-api-access-22qrq\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676865 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-host\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676869 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-registration-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676879 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-node-log\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676887 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-run\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676931 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676937 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-host\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676946 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676956 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-sys-fs\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676986 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-run\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.676980 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-systemd-units\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677004 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.677695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-sys-fs\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677028 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-systemd-units\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-netns\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677076 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677085 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jtl\" (UniqueName: \"kubernetes.io/projected/50406e5f-7f63-49e5-aca1-d572b155292a-kube-api-access-f5jtl\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677114 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677121 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-netns\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-config\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-kubelet-config\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677170 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.677237 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-kubelet-config\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:39.678332 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.678305 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-tmp\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.678696 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.678680 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-etc-tuned\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.678880 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.678865 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-agent-certs\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.684740 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684693 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:45:39.684740 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684741 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pvjtk\"" Oct 30 21:45:39.684948 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684753 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rn9hr\"" Oct 30 21:45:39.684948 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684783 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fd5qx\"" Oct 30 21:45:39.684948 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684794 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bg2hw\"" Oct 30 21:45:39.684948 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684890 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:45:39.684948 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684907 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mkvf\"" Oct 30 21:45:39.684948 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684942 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9dfpb\"" Oct 30 21:45:39.685210 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.684959 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:45:39.685210 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685025 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqqg2\"" Oct 30 21:45:39.685210 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685077 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5wwct\"" Oct 30 21:45:39.685499 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685485 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.685661 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685637 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.685661 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685637 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.685766 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685717 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:45:39.685766 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685750 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.685981 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685767 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.685981 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685787 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.685981 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685798 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.685981 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685808 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:45:39.685981 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685824 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:45:39.685981 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.685945 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686010 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686017 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686030 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686038 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686068 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686075 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686082 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686163 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686199 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-cni-binary-copy\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686230 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686257 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.686370 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.686360 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.687725 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.687710 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-env-overrides\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.687797 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.687772 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-config\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.687919 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.687902 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/552b3503-9502-4ac7-af98-0acbc947254c-iptables-alerter-script\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.687952 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.687914 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.693458 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.693442 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b9b148-13a4-40f3-9113-2877f5ca9574-serviceca\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.693540 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.693463 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-daemon-config\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.693770 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.693754 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-script-lib\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.695193 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.695175 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovn-node-metrics-cert\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.695377 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.695360 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.695441 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.695365 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.730865 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.730843 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58vn\" (UniqueName: \"kubernetes.io/projected/16867cf6-0526-47b2-866e-5dfc13f04869-kube-api-access-l58vn\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.731815 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.731791 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkccl\" (UniqueName: \"kubernetes.io/projected/f008c236-2573-46f0-bbdd-f10daa5d08bf-kube-api-access-lkccl\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.731947 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.731932 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82np\" (UniqueName: \"kubernetes.io/projected/552b3503-9502-4ac7-af98-0acbc947254c-kube-api-access-d82np\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.732289 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.732275 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4qt\" (UniqueName: \"kubernetes.io/projected/5af23954-d911-4664-872d-9ae2e67ecfa0-kube-api-access-cw4qt\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:45:39.735239 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.735224 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtl9\" (UniqueName: \"kubernetes.io/projected/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kube-api-access-vjtl9\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.735976 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.735963 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:39.736029 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.735980 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:39.736029 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.735990 2571 projected.go:196] Error preparing data for projected volume kube-api-access-dzk64 for pod openshift-network-diagnostics/network-check-target-tqcc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:39.736090 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:39.736040 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64 podName:573e461a-8123-4e53-9134-80d6eb9a3e71 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:40.236025929 +0000 UTC m=+3.199630356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dzk64" (UniqueName: "kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64") pod "network-check-target-tqcc7" (UID: "573e461a-8123-4e53-9134-80d6eb9a3e71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:39.739446 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.739425 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jtl\" (UniqueName: \"kubernetes.io/projected/50406e5f-7f63-49e5-aca1-d572b155292a-kube-api-access-f5jtl\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.740387 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.740363 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47qr\" (UniqueName: \"kubernetes.io/projected/12b9b148-13a4-40f3-9113-2877f5ca9574-kube-api-access-j47qr\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.742553 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.742536 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qrq\" (UniqueName: \"kubernetes.io/projected/dbdaddf0-875f-46f2-8418-62377b63a086-kube-api-access-22qrq\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:39.743154 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.743133 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntnrc\" (UniqueName: \"kubernetes.io/projected/2a5249fa-084f-451b-8e94-9cbd497a8ea7-kube-api-access-ntnrc\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.846535 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.846509 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:39.853597 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.853576 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:45:39.856185 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.856163 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad6427c_0014_4c83_9ff9_ce1d9ba0aa55.slice/crio-2e08256481da327cead75d4879ec52c69fe23a634c3bd64872cc01c3014958c0 WatchSource:0}: Error finding container 2e08256481da327cead75d4879ec52c69fe23a634c3bd64872cc01c3014958c0: Status 404 returned error can't find the container with id 2e08256481da327cead75d4879ec52c69fe23a634c3bd64872cc01c3014958c0 Oct 30 21:45:39.861190 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.861173 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-746ws" Oct 30 21:45:39.861451 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.861432 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b9b148_13a4_40f3_9113_2877f5ca9574.slice/crio-1626321413d45115210cddc16a6e7171d7331e29e6219f828300aece3186d985 WatchSource:0}: Error finding container 1626321413d45115210cddc16a6e7171d7331e29e6219f828300aece3186d985: Status 404 returned error can't find the container with id 1626321413d45115210cddc16a6e7171d7331e29e6219f828300aece3186d985 Oct 30 21:45:39.865700 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.865684 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:45:39.869793 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.869775 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af23954_d911_4664_872d_9ae2e67ecfa0.slice/crio-509027a15f063ed048e5d0e6cebbc821d5a4e612539b315bc5c57591aeb57ffe WatchSource:0}: Error finding container 509027a15f063ed048e5d0e6cebbc821d5a4e612539b315bc5c57591aeb57ffe: Status 404 returned error can't find the container with id 509027a15f063ed048e5d0e6cebbc821d5a4e612539b315bc5c57591aeb57ffe Oct 30 21:45:39.870244 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.870227 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g66br" Oct 30 21:45:39.874185 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.874169 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552b3503_9502_4ac7_af98_0acbc947254c.slice/crio-1e8e18cada4240db6dc7842e0cdca1e77cc61bc41b3c961c8425b94e068e030d WatchSource:0}: Error finding container 1e8e18cada4240db6dc7842e0cdca1e77cc61bc41b3c961c8425b94e068e030d: Status 404 returned error can't find the container with id 1e8e18cada4240db6dc7842e0cdca1e77cc61bc41b3c961c8425b94e068e030d Oct 30 21:45:39.878768 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.878750 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16867cf6_0526_47b2_866e_5dfc13f04869.slice/crio-961818bf1058c55fa1d060bb4f59e064c43d1e75c30844cfa257aeb53528aaf7 WatchSource:0}: Error finding container 961818bf1058c55fa1d060bb4f59e064c43d1e75c30844cfa257aeb53528aaf7: Status 404 returned error can't find the container with id 961818bf1058c55fa1d060bb4f59e064c43d1e75c30844cfa257aeb53528aaf7 Oct 30 21:45:39.880311 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.880295 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:45:39.882618 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.882603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:45:39.889989 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.889969 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b2124f_2877_4c7b_8fc1_75d3bda8c459.slice/crio-88f30cdce0e32ff2eb885806ed2cd6a8952f5711092d6a8f84f75fd9c5e7f073 WatchSource:0}: Error finding container 88f30cdce0e32ff2eb885806ed2cd6a8952f5711092d6a8f84f75fd9c5e7f073: Status 404 returned error can't find the container with id 88f30cdce0e32ff2eb885806ed2cd6a8952f5711092d6a8f84f75fd9c5e7f073 Oct 30 21:45:39.890599 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.890577 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50406e5f_7f63_49e5_aca1_d572b155292a.slice/crio-eec44ea7f50fd91c15faa2726b44fd373f3f992bd63570a3841167f999294d11 WatchSource:0}: Error finding container eec44ea7f50fd91c15faa2726b44fd373f3f992bd63570a3841167f999294d11: Status 404 returned error can't find the container with id eec44ea7f50fd91c15faa2726b44fd373f3f992bd63570a3841167f999294d11 Oct 30 21:45:39.903747 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.903733 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:45:39.912174 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.912151 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5249fa_084f_451b_8e94_9cbd497a8ea7.slice/crio-7793ccc2ca8bd97467b6023a3f53c8abaa3350e83a2e74fa7ca0c53a95649409 WatchSource:0}: Error finding container 7793ccc2ca8bd97467b6023a3f53c8abaa3350e83a2e74fa7ca0c53a95649409: Status 404 returned error can't find the container with id 7793ccc2ca8bd97467b6023a3f53c8abaa3350e83a2e74fa7ca0c53a95649409 Oct 30 21:45:39.915133 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:39.915119 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:39.923178 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:45:39.923162 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf008c236_2573_46f0_bbdd_f10daa5d08bf.slice/crio-7525d5af9e663240e9512b9842541a26515277797825ebd2acc26ed5300509b3 WatchSource:0}: Error finding container 7525d5af9e663240e9512b9842541a26515277797825ebd2acc26ed5300509b3: Status 404 returned error can't find the container with id 7525d5af9e663240e9512b9842541a26515277797825ebd2acc26ed5300509b3 Oct 30 21:45:40.181243 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.181183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:40.181243 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.181216 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:40.181424 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.181316 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:40.181424 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.181328 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:40.181424 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.181375 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret podName:d3280f18-2d95-4303-8f34-5a2fe91671e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:41.181359368 +0000 UTC m=+4.144963785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret") pod "global-pull-secret-syncer-4zhvv" (UID: "d3280f18-2d95-4303-8f34-5a2fe91671e6") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:40.181424 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.181390 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:41.18138297 +0000 UTC m=+4.144987380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:40.282104 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.282076 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:40.282281 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.282206 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:40.282281 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.282226 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:40.282281 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.282237 2571 projected.go:196] Error preparing data for projected volume kube-api-access-dzk64 for pod openshift-network-diagnostics/network-check-target-tqcc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:40.282506 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:40.282295 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64 podName:573e461a-8123-4e53-9134-80d6eb9a3e71 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:41.282273914 +0000 UTC m=+4.245878325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzk64" (UniqueName: "kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64") pod "network-check-target-tqcc7" (UID: "573e461a-8123-4e53-9134-80d6eb9a3e71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:40.529516 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.529483 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"7525d5af9e663240e9512b9842541a26515277797825ebd2acc26ed5300509b3"} Oct 30 21:45:40.539306 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.539274 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerStarted","Data":"7793ccc2ca8bd97467b6023a3f53c8abaa3350e83a2e74fa7ca0c53a95649409"} Oct 30 21:45:40.540462 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.540427 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" event={"ID":"50406e5f-7f63-49e5-aca1-d572b155292a","Type":"ContainerStarted","Data":"eec44ea7f50fd91c15faa2726b44fd373f3f992bd63570a3841167f999294d11"} Oct 30 21:45:40.541674 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.541652 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" event={"ID":"d9b2124f-2877-4c7b-8fc1-75d3bda8c459","Type":"ContainerStarted","Data":"88f30cdce0e32ff2eb885806ed2cd6a8952f5711092d6a8f84f75fd9c5e7f073"} Oct 30 21:45:40.542954 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.542930 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g66br" event={"ID":"16867cf6-0526-47b2-866e-5dfc13f04869","Type":"ContainerStarted","Data":"961818bf1058c55fa1d060bb4f59e064c43d1e75c30844cfa257aeb53528aaf7"} Oct 30 21:45:40.544723 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.544699 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fmwmz" event={"ID":"552b3503-9502-4ac7-af98-0acbc947254c","Type":"ContainerStarted","Data":"1e8e18cada4240db6dc7842e0cdca1e77cc61bc41b3c961c8425b94e068e030d"} Oct 30 21:45:40.546236 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.546214 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r2dj5" event={"ID":"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55","Type":"ContainerStarted","Data":"2e08256481da327cead75d4879ec52c69fe23a634c3bd64872cc01c3014958c0"} Oct 30 21:45:40.547816 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.547794 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-746ws" event={"ID":"5af23954-d911-4664-872d-9ae2e67ecfa0","Type":"ContainerStarted","Data":"509027a15f063ed048e5d0e6cebbc821d5a4e612539b315bc5c57591aeb57ffe"} Oct 30 21:45:40.549171 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.549147 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xm8l9" event={"ID":"12b9b148-13a4-40f3-9113-2877f5ca9574","Type":"ContainerStarted","Data":"1626321413d45115210cddc16a6e7171d7331e29e6219f828300aece3186d985"} Oct 30 21:45:40.632800 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.632734 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:39 +0000 UTC" deadline="2028-04-18 06:13:21.25784984 +0000 UTC" Oct 30 21:45:40.632800 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:40.632798 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12848h27m40.625055516s" Oct 30 21:45:41.191458 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.191425 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:41.191616 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.191478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:41.191616 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.191604 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:41.191759 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.191659 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret podName:d3280f18-2d95-4303-8f34-5a2fe91671e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:43.191642438 +0000 UTC m=+6.155246854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret") pod "global-pull-secret-syncer-4zhvv" (UID: "d3280f18-2d95-4303-8f34-5a2fe91671e6") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:41.191759 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.191721 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:41.191759 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.191757 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:43.191745215 +0000 UTC m=+6.155349628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:41.292421 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.292390 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:41.292618 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.292540 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:41.292618 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.292561 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:41.292618 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.292572 2571 projected.go:196] Error preparing data for projected volume kube-api-access-dzk64 for pod openshift-network-diagnostics/network-check-target-tqcc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:41.292796 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.292624 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64 podName:573e461a-8123-4e53-9134-80d6eb9a3e71 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:43.292607487 +0000 UTC m=+6.256211911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzk64" (UniqueName: "kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64") pod "network-check-target-tqcc7" (UID: "573e461a-8123-4e53-9134-80d6eb9a3e71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:41.524245 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.524213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:41.524435 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.524365 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:41.524512 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.524446 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:41.524571 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.524520 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:41.524626 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.524585 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:41.524679 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:41.524652 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:41.633313 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.633274 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:39 +0000 UTC" deadline="2028-07-01 18:18:32.583351666 +0000 UTC" Oct 30 21:45:41.633313 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:41.633308 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14636h32m50.950047294s" Oct 30 21:45:43.210023 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.209761 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:43.210023 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.209985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:43.210692 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.210122 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:43.210692 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.210199 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret podName:d3280f18-2d95-4303-8f34-5a2fe91671e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:47.210179825 +0000 UTC m=+10.173784249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret") pod "global-pull-secret-syncer-4zhvv" (UID: "d3280f18-2d95-4303-8f34-5a2fe91671e6") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:43.210692 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.210270 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:43.210692 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.210301 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:47.210290932 +0000 UTC m=+10.173895357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:43.311358 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.311332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:43.311459 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.311440 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:43.311525 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.311461 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:43.311525 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.311473 2571 projected.go:196] Error preparing data for projected volume kube-api-access-dzk64 for pod openshift-network-diagnostics/network-check-target-tqcc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:43.311525 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.311522 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64 podName:573e461a-8123-4e53-9134-80d6eb9a3e71 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:47.31150402 +0000 UTC m=+10.275108444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzk64" (UniqueName: "kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64") pod "network-check-target-tqcc7" (UID: "573e461a-8123-4e53-9134-80d6eb9a3e71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:43.524797 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.524360 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:43.524797 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.524488 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:43.524797 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.524572 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:43.524797 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.524641 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:43.524797 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.524698 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:43.524797 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:43.524758 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:43.577236 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.577201 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"44d4a0aec485b9ad44110c51253a50d0","Type":"ContainerStarted","Data":"d7d72df3f2394312417c189db556e7a76a43dc9b2ed9283b8385819ae1743523"} Oct 30 21:45:43.585210 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.585181 2571 generic.go:356] "Generic (PLEG): container finished" podID="fdf14b302ecfd0d0d1206ce2297bec21" containerID="9a1ef903ad4537dfc212d0dda17fed4d4a4c9e651874eb3e4ea06a078d15b4f6" exitCode=0 Oct 30 21:45:43.585340 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.585272 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"fdf14b302ecfd0d0d1206ce2297bec21","Type":"ContainerDied","Data":"9a1ef903ad4537dfc212d0dda17fed4d4a4c9e651874eb3e4ea06a078d15b4f6"} Oct 30 21:45:43.812338 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:43.808707 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" podStartSLOduration=5.808687015 podStartE2EDuration="5.808687015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:43.688679162 +0000 UTC m=+6.652283595" watchObservedRunningTime="2026-10-30 21:45:43.808687015 +0000 UTC m=+6.772291449" Oct 30 21:45:44.601291 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:44.600584 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"fdf14b302ecfd0d0d1206ce2297bec21","Type":"ContainerStarted","Data":"36b378df4cdc8bde691544927d8a7f939cdaa05f53238fb924f3b12b0564167d"} Oct 30 21:45:44.608158 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:44.607725 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="c8793d0efe53fd42b7d407508e6974a79e72c9634956751c25b5e5b09a95a099" exitCode=0 Oct 30 21:45:44.608158 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:44.607901 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerDied","Data":"c8793d0efe53fd42b7d407508e6974a79e72c9634956751c25b5e5b09a95a099"} Oct 30 21:45:44.714308 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:44.713179 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" podStartSLOduration=6.71316185 podStartE2EDuration="6.71316185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:44.712939643 +0000 UTC m=+7.676544076" watchObservedRunningTime="2026-10-30 21:45:44.71316185 +0000 UTC m=+7.676766285" Oct 30 21:45:45.524568 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:45.524051 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:45.524568 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:45.524190 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:45.524568 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:45.524272 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:45.524568 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:45.524374 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:45.524568 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:45.524436 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:45.524568 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:45.524494 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:45.613455 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:45.613393 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r2dj5" event={"ID":"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55","Type":"ContainerStarted","Data":"98a3c3ef9ef8b7d1b1257fa47c725e0bb5654b3edb1fd7d9a94ab04bd4daa529"} Oct 30 21:45:45.726721 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:45.726661 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r2dj5" podStartSLOduration=3.162501964 podStartE2EDuration="7.726642982s" totalImagesPullingTime="4.564141018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:45.725768064 +0000 UTC m=+8.689372498" watchObservedRunningTime="2026-10-30 21:45:45.726642982 +0000 UTC m=+8.690247417" Oct 30 21:45:46.617569 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:46.617522 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" event={"ID":"d9b2124f-2877-4c7b-8fc1-75d3bda8c459","Type":"ContainerStarted","Data":"03c5aa91f07c9820924dcae33a40d8f7fa050c47729d6717b8bb94184e42cfab"} Oct 30 21:45:46.623882 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:46.623856 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xm8l9" event={"ID":"12b9b148-13a4-40f3-9113-2877f5ca9574","Type":"ContainerStarted","Data":"8adb71e382c2670071cd44c318f974eec337d57ff7632e16a244c812e3acbe3c"} Oct 30 21:45:47.248901 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:47.248859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:47.249079 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:47.248925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:47.249140 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.249086 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:47.249192 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.249145 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret podName:d3280f18-2d95-4303-8f34-5a2fe91671e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:55.249126069 +0000 UTC m=+18.212730497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret") pod "global-pull-secret-syncer-4zhvv" (UID: "d3280f18-2d95-4303-8f34-5a2fe91671e6") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:47.249636 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.249615 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:47.249708 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.249679 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:55.249662169 +0000 UTC m=+18.213266595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:47.349582 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:47.349550 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:47.349733 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.349718 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:47.349772 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.349743 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:47.349772 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.349769 2571 projected.go:196] Error preparing data for projected volume kube-api-access-dzk64 for pod openshift-network-diagnostics/network-check-target-tqcc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:47.349833 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.349823 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64 podName:573e461a-8123-4e53-9134-80d6eb9a3e71 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:55.349804625 +0000 UTC m=+18.313409041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzk64" (UniqueName: "kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64") pod "network-check-target-tqcc7" (UID: "573e461a-8123-4e53-9134-80d6eb9a3e71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:47.525592 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:47.525523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:47.525877 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.525853 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:47.525997 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:47.525716 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:47.526219 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:47.525698 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:47.526350 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.526311 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:47.526417 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:47.526198 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:48.208630 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.208597 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:45:48.525347 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.525231 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:45:48.208617994Z","UUID":"897d66ab-f37c-4f8c-aac0-01fbb891bead","Handler":null,"Name":"","Endpoint":""} Oct 30 21:45:48.526763 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.526728 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:45:48.526763 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.526756 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:45:48.633384 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.629793 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" event={"ID":"d9b2124f-2877-4c7b-8fc1-75d3bda8c459","Type":"ContainerStarted","Data":"be08d39849b62f724233830d4ba7c3822e8bba5e1e232126c0d161547cae66f8"} Oct 30 21:45:48.633384 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.631689 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-746ws" event={"ID":"5af23954-d911-4664-872d-9ae2e67ecfa0","Type":"ContainerStarted","Data":"bbfad514bf65087affe960296a502a99cd75e72673710017161d6125b93d2614"} Oct 30 21:45:48.763717 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.763610 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xm8l9" podStartSLOduration=4.903727281 podStartE2EDuration="10.763592584s" totalImagesPullingTime="5.859865303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:46.733936234 +0000 UTC m=+9.697540668" watchObservedRunningTime="2026-10-30 21:45:48.763592584 +0000 UTC m=+11.727197018" Oct 30 21:45:48.763855 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.763714 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-746ws" podStartSLOduration=2.7966238199999998 podStartE2EDuration="10.763706975s" totalImagesPullingTime="7.967083155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:48.761315566 +0000 UTC m=+11.724920004" watchObservedRunningTime="2026-10-30 21:45:48.763706975 +0000 UTC m=+11.727311409" Oct 30 21:45:48.886594 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.886534 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:48.887103 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:48.887082 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:49.521821 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:49.521789 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:49.522269 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:49.521937 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:49.522358 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:49.522292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:49.522433 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:49.522404 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:49.522483 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:49.522468 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:49.522538 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:49.522521 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:49.637920 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:49.637892 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="bf9102c6954d81fc3611860cc61d08aa1295c4d6c31aa5fdbac120a18e280954" exitCode=0 Oct 30 21:45:49.638011 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:49.637981 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerDied","Data":"bf9102c6954d81fc3611860cc61d08aa1295c4d6c31aa5fdbac120a18e280954"} Oct 30 21:45:49.642032 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:49.641990 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g66br" event={"ID":"16867cf6-0526-47b2-866e-5dfc13f04869","Type":"ContainerStarted","Data":"dcc34cbfee80c363a8e009b6e5f487dbd137c8b85008324aa1aff2823d965676"} Oct 30 21:45:49.865293 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:49.864990 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-g66br" podStartSLOduration=3.24978961 podStartE2EDuration="11.864960946s" totalImagesPullingTime="8.615171336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:49.864269333 +0000 UTC m=+12.827873768" watchObservedRunningTime="2026-10-30 21:45:49.864960946 +0000 UTC m=+12.828565382" Oct 30 21:45:50.647586 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:50.647492 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" event={"ID":"50406e5f-7f63-49e5-aca1-d572b155292a","Type":"ContainerStarted","Data":"9996181d03283571cb0cf1060b0e140e3e8cbf92869d5a5aac2df664129e2737"} Oct 30 21:45:50.650113 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:50.650083 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" event={"ID":"d9b2124f-2877-4c7b-8fc1-75d3bda8c459","Type":"ContainerStarted","Data":"b62b9f73cfe33236801a32f020b945f52ced44e5f480a8676dd18c8ffcce136e"} Oct 30 21:45:50.652112 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:50.652087 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fmwmz" event={"ID":"552b3503-9502-4ac7-af98-0acbc947254c","Type":"ContainerStarted","Data":"a8bf8fd60cd20c2f1eae652793a9053c0c3c93dec5d5c51546e9d01483147a5a"} Oct 30 21:45:50.762494 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:50.762442 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" podStartSLOduration=2.153049868 podStartE2EDuration="12.762424289s" totalImagesPullingTime="10.609374421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:50.762033522 +0000 UTC m=+13.725637957" watchObservedRunningTime="2026-10-30 21:45:50.762424289 +0000 UTC m=+13.726028701" Oct 30 21:45:50.882991 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:50.882928 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fmwmz" podStartSLOduration=4.480427595 podStartE2EDuration="12.882909133s" totalImagesPullingTime="8.402481538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:50.882460644 +0000 UTC m=+13.846065076" watchObservedRunningTime="2026-10-30 21:45:50.882909133 +0000 UTC m=+13.846513568" Oct 30 21:45:51.521531 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:51.521499 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:51.521689 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:51.521504 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:51.521689 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:51.521636 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:51.521810 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:51.521608 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:51.521810 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:51.521717 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:51.521810 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:51.521802 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:51.656150 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:51.656115 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="32e430384c42f8f98627988b1d4fd9942d5740648679194e56c9b5b666ed1aa9" exitCode=0 Oct 30 21:45:51.656873 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:51.656158 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerDied","Data":"32e430384c42f8f98627988b1d4fd9942d5740648679194e56c9b5b666ed1aa9"} Oct 30 21:45:51.775219 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:51.775121 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" podStartSLOduration=3.640539397 podStartE2EDuration="13.775101047s" totalImagesPullingTime="10.13456165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:50.998205233 +0000 UTC m=+13.961809665" watchObservedRunningTime="2026-10-30 21:45:51.775101047 +0000 UTC m=+14.738705483" Oct 30 21:45:53.522239 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:53.521977 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:53.522757 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:53.521982 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:53.522757 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:53.522357 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:53.522757 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:53.522041 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:53.522757 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:53.522443 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:53.522757 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:53.522521 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:53.663044 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:53.663013 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="b64f9bd6bf2323ac8b0f2a231af9cfaf29f8ea768a1b4750eafa996d82830ab4" exitCode=0 Oct 30 21:45:53.663204 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:53.663054 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerDied","Data":"b64f9bd6bf2323ac8b0f2a231af9cfaf29f8ea768a1b4750eafa996d82830ab4"} Oct 30 21:45:55.310487 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.310458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:55.310877 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.310609 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:55.310877 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.310659 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.310645276 +0000 UTC m=+34.274249700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:55.310877 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.310606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:55.310877 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.310686 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:55.310877 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.310743 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret podName:d3280f18-2d95-4303-8f34-5a2fe91671e6 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.31072745 +0000 UTC m=+34.274331876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret") pod "global-pull-secret-syncer-4zhvv" (UID: "d3280f18-2d95-4303-8f34-5a2fe91671e6") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:55.412113 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.412083 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:55.412219 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.412201 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:55.412273 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.412222 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:55.412273 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.412232 2571 projected.go:196] Error preparing data for projected volume kube-api-access-dzk64 for pod openshift-network-diagnostics/network-check-target-tqcc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:55.412369 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.412277 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64 podName:573e461a-8123-4e53-9134-80d6eb9a3e71 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.412261163 +0000 UTC m=+34.375865574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzk64" (UniqueName: "kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64") pod "network-check-target-tqcc7" (UID: "573e461a-8123-4e53-9134-80d6eb9a3e71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:55.522422 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.522201 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:55.522556 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.522536 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:55.522630 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.522616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:55.522714 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.522697 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:55.522797 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.522785 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:55.522880 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:55.522865 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:55.670390 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.670299 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"e0d74f661609c5f8e5cfe8ab78316eb369ef783fb383ddc9fc100d747018031f"} Oct 30 21:45:55.670529 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.670403 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"aece648b6759e098a59b15acc307504ea6fcb95fb3be3f245f35f77b44ce4a08"} Oct 30 21:45:55.670529 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.670459 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"c0027c77ab8ae14fa649dd1aa91037edd8649fd0d653549c6ff23b5d95364f8f"} Oct 30 21:45:55.670529 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.670503 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"c23106772bad222449218a50b52f27a6e6635861888e9f5abb33902a655a07ad"} Oct 30 21:45:55.670695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.670549 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"9db68523c1d0ac5a0a65afca2834b33fb93f21926c3d979557afd0d807c6be08"} Oct 30 21:45:55.670695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:55.670594 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"0042b6db18e3383e66cd42e9a2c0fa8f1b600364f8dcc917bccf98432c3a6793"} Oct 30 21:45:56.674631 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:56.674601 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="debc53a084c45e1711152d4542e379fc44b804689b1498196f156852beba9a87" exitCode=0 Oct 30 21:45:56.675029 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:56.674649 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerDied","Data":"debc53a084c45e1711152d4542e379fc44b804689b1498196f156852beba9a87"} Oct 30 21:45:57.522363 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:57.522334 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:57.522549 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:57.522443 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:57.522549 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:57.522477 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:57.522549 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:57.522524 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:57.522696 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:57.522579 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:57.522696 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:57.522664 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:57.680592 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:57.680560 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"75a594748c6d55b6b1f2d0f54dda9cea68df0c04d0018314077f42b2bde15922"} Oct 30 21:45:57.683725 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:57.683699 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="b48f5a4ff29408b4e28dda7e4b6704b1be8eb57707c6ebb9ca708056f6c6ebcc" exitCode=0 Oct 30 21:45:57.683829 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:57.683747 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerDied","Data":"b48f5a4ff29408b4e28dda7e4b6704b1be8eb57707c6ebb9ca708056f6c6ebcc"} Oct 30 21:45:58.254221 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:58.254184 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:58.254406 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:58.254340 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:45:58.254875 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:58.254859 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:45:58.688575 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:58.688492 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznvs" event={"ID":"2a5249fa-084f-451b-8e94-9cbd497a8ea7","Type":"ContainerStarted","Data":"aa4b0bc422603e184565d3377d5905b2681891b5d224384528cab73a3d59a912"} Oct 30 21:45:58.806421 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:58.806373 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vznvs" podStartSLOduration=7.82736859 podStartE2EDuration="20.806360582s" totalImagesPullingTime="12.978991992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:58.805912904 +0000 UTC m=+21.769517334" watchObservedRunningTime="2026-10-30 21:45:58.806360582 +0000 UTC m=+21.769965015" Oct 30 21:45:59.524361 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:59.524333 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:45:59.524361 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:59.524349 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:45:59.524361 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:59.524343 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:45:59.524575 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:59.524433 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:45:59.524575 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:59.524484 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:45:59.524575 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:45:59.524541 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:45:59.694545 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:59.694511 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" event={"ID":"f008c236-2573-46f0-bbdd-f10daa5d08bf","Type":"ContainerStarted","Data":"162a91fc0ec21f3b46badf15d93a6deb95e32ba0492347057db665ff047f4331"} Oct 30 21:45:59.695209 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:59.694750 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:59.709569 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:59.709547 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:45:59.936979 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:45:59.936755 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" podStartSLOduration=6.701745026 podStartE2EDuration="21.936744065s" totalImagesPullingTime="15.234999039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:59.936614543 +0000 UTC m=+22.900218976" watchObservedRunningTime="2026-10-30 21:45:59.936744065 +0000 UTC m=+22.900348497" Oct 30 21:46:00.697094 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:00.697066 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:00.697504 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:00.697168 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:00.712219 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:00.712198 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:01.522083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:01.522050 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:01.522344 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:01.522059 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:01.522344 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:01.522167 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:46:01.522344 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:01.522176 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:01.522344 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:01.522235 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:46:01.522344 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:01.522332 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:46:02.337470 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:02.337436 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqcc7"] Oct 30 21:46:02.337890 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:02.337529 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:02.337890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:02.337608 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:46:02.340181 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:02.340159 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zhvv"] Oct 30 21:46:02.340313 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:02.340234 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:02.340313 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:02.340304 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:46:02.342866 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:02.342844 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbmj9"] Oct 30 21:46:02.342944 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:02.342918 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:02.343008 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:02.342990 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:46:03.525004 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:03.524975 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:03.525403 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:03.525071 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:46:04.522265 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:04.522235 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:04.522436 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:04.522235 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:04.522436 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:04.522355 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:46:04.522436 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:04.522401 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:46:05.521597 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:05.521568 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:05.521990 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:05.521662 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:46:06.521838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:06.521806 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:06.521838 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:06.521819 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:06.522355 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:06.521935 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbmj9" podUID="dbdaddf0-875f-46f2-8418-62377b63a086" Oct 30 21:46:06.522355 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:06.522032 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zhvv" podUID="d3280f18-2d95-4303-8f34-5a2fe91671e6" Oct 30 21:46:07.528104 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.528081 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:07.528532 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:07.528172 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqcc7" podUID="573e461a-8123-4e53-9134-80d6eb9a3e71" Oct 30 21:46:07.812269 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.812218 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeReady" Oct 30 21:46:07.812379 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.812309 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:46:07.922285 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.922255 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p2dr2"] Oct 30 21:46:07.926740 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.926723 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dll8d"] Oct 30 21:46:07.926866 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.926846 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" Oct 30 21:46:07.931915 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.931897 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2jxfz"] Oct 30 21:46:07.932025 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.932010 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dll8d" Oct 30 21:46:07.934958 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.934939 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw"] Oct 30 21:46:07.935081 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.935063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" Oct 30 21:46:07.938496 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.938477 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p"] Oct 30 21:46:07.938585 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.938572 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" Oct 30 21:46:07.941257 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.941239 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:07.955957 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.955941 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c"] Oct 30 21:46:07.959288 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.959272 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p"] Oct 30 21:46:07.959418 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.959402 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:07.962586 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.962567 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:07.971458 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.970368 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jqbm4"] Oct 30 21:46:07.973678 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.973660 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j66r8"] Oct 30 21:46:07.973766 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.973728 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:07.976945 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.976927 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58d68ffbc4-r5mnd"] Oct 30 21:46:07.977048 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.977033 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:07.980273 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.980129 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tc7w4"] Oct 30 21:46:07.980273 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.980242 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:07.982539 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.982522 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d"] Oct 30 21:46:07.982635 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.982622 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:07.985311 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.985292 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c8c9db44f-llgx4"] Oct 30 21:46:07.985417 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.985400 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:07.989091 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.989074 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b65fb6c59-872xp"] Oct 30 21:46:07.989206 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.989191 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:07.991384 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.991366 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr"] Oct 30 21:46:07.991470 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.991410 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:07.994680 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.994665 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8fwrp"] Oct 30 21:46:07.994777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.994763 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:07.998135 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qdqzz"] Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.013264 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.013842 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p2dr2"] Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.013923 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.015186 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45q9q\" (UniqueName: \"kubernetes.io/projected/c6eb7759-f63d-4817-9128-adcecdceb0d7-kube-api-access-45q9q\") pod \"volume-data-source-validator-58c9f5f46d-h6cxw\" (UID: \"c6eb7759-f63d-4817-9128-adcecdceb0d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.015217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7mx\" (UniqueName: \"kubernetes.io/projected/177b7d0a-d868-497a-b648-23228667c743-kube-api-access-7z7mx\") pod \"kas-connection-checker-685895c7c9-dll8d\" (UID: \"177b7d0a-d868-497a-b648-23228667c743\") " pod="kube-system/kas-connection-checker-685895c7c9-dll8d" Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.015299 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7xlh\" (UniqueName: \"kubernetes.io/projected/0abc3e92-4ede-4421-ab8d-91477bd6b85e-kube-api-access-r7xlh\") pod \"kas-connection-checker-685895c7c9-p2dr2\" (UID: \"0abc3e92-4ede-4421-ab8d-91477bd6b85e\") " pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" Oct 30 21:46:08.016346 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.015593 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxtvm\" (UniqueName: \"kubernetes.io/projected/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a-kube-api-access-nxtvm\") pod \"kas-connection-checker-685895c7c9-2jxfz\" (UID: \"6ffd4edc-ce40-4a48-b398-c2d907a7cd0a\") " pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" Oct 30 21:46:08.019258 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.017416 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:46:08.019258 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.017419 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.019258 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.017494 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.019258 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.017546 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.019258 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.019201 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dll8d"] Oct 30 21:46:08.019754 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.019390 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.020437 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.020416 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p"] Oct 30 21:46:08.021279 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.021257 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2jxfz"] Oct 30 21:46:08.033805 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.033785 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p"] Oct 30 21:46:08.035017 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.034990 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c"] Oct 30 21:46:08.035451 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.035432 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jqbm4"] Oct 30 21:46:08.036053 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.036035 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j66r8"] Oct 30 21:46:08.043891 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.043710 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d68ffbc4-r5mnd"] Oct 30 21:46:08.043891 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.043754 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tc7w4"] Oct 30 21:46:08.044947 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.044929 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw"] Oct 30 21:46:08.081954 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.081935 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.082048 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082023 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.082048 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082029 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.082311 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082291 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.082455 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082347 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.082455 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082364 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.082455 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082425 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.082455 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082431 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.082455 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082433 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:46:08.082742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082555 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:46:08.082742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082556 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.082742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082583 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.082742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082606 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:46:08.082742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082658 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:46:08.082742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082712 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:46:08.082742 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082734 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.083153 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.082974 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.083786 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.083770 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.083923 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.083898 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:46:08.084839 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.084752 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b65fb6c59-872xp"] Oct 30 21:46:08.088684 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.088668 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:46:08.116379 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116356 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:08.116468 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116386 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.116468 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116420 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.116534 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116478 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a07200b0-a7f6-4c5a-9197-b73779a06c1f-serving-cert\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.116534 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116503 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.116695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116535 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.116695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.116695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116568 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7xlh\" (UniqueName: \"kubernetes.io/projected/0abc3e92-4ede-4421-ab8d-91477bd6b85e-kube-api-access-r7xlh\") pod \"kas-connection-checker-685895c7c9-p2dr2\" (UID: \"0abc3e92-4ede-4421-ab8d-91477bd6b85e\") " pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" Oct 30 21:46:08.116695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116583 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxtvm\" (UniqueName: \"kubernetes.io/projected/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a-kube-api-access-nxtvm\") pod \"kas-connection-checker-685895c7c9-2jxfz\" (UID: \"6ffd4edc-ce40-4a48-b398-c2d907a7cd0a\") " pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" Oct 30 21:46:08.116695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.116835 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqdn\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.116835 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116750 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sz9x\" (UniqueName: \"kubernetes.io/projected/2da4fada-55a5-42ad-b873-46f86929c5d0-kube-api-access-5sz9x\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.116835 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6594a6f-9c78-40bf-99b7-fb21cd031157-serving-cert\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.116927 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvtf\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.116927 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116872 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3afb35-bfde-45ec-a972-78a6e7461d9d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.116927 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04efc6e4-19af-4935-a7c1-2dac67498e4c-serving-cert\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.116927 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwfx\" (UniqueName: \"kubernetes.io/projected/4f3afb35-bfde-45ec-a972-78a6e7461d9d-kube-api-access-thwfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.117038 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-snapshots\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.117038 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116951 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcqn\" (UniqueName: \"kubernetes.io/projected/a07200b0-a7f6-4c5a-9197-b73779a06c1f-kube-api-access-rpcqn\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.117038 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.116976 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.117038 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117009 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.117038 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117032 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-default-certificate\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.117175 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117054 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqg42\" (UniqueName: \"kubernetes.io/projected/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-kube-api-access-cqg42\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.117175 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117092 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-tmp\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.117175 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117110 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7mx\" (UniqueName: \"kubernetes.io/projected/177b7d0a-d868-497a-b648-23228667c743-kube-api-access-7z7mx\") pod \"kas-connection-checker-685895c7c9-dll8d\" (UID: \"177b7d0a-d868-497a-b648-23228667c743\") " pod="kube-system/kas-connection-checker-685895c7c9-dll8d" Oct 30 21:46:08.117175 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117126 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8743ad27-5d90-4b7f-8012-6faf21548f97-config-volume\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.117175 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117143 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.117175 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd5v\" (UniqueName: \"kubernetes.io/projected/7e37f363-3bfe-4f18-a5ee-f60f849499d1-kube-api-access-7bd5v\") pod \"network-check-source-6f78d59f99-qzvdr\" (UID: \"7e37f363-3bfe-4f18-a5ee-f60f849499d1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" Oct 30 21:46:08.117175 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxh4\" (UniqueName: \"kubernetes.io/projected/6b4c949e-cff6-4abf-b3ff-f85046f76784-kube-api-access-2pxh4\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.117402 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117216 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.117402 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.117402 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117293 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-config\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.117402 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117352 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.117402 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117382 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.117539 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.117539 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.117539 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117457 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.117539 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117483 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2xf\" (UniqueName: \"kubernetes.io/projected/8743ad27-5d90-4b7f-8012-6faf21548f97-kube-api-access-7s2xf\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.117539 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117498 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqd5\" (UniqueName: \"kubernetes.io/projected/d6594a6f-9c78-40bf-99b7-fb21cd031157-kube-api-access-rtqd5\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.117675 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117541 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.117675 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da4fada-55a5-42ad-b873-46f86929c5d0-cert\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.117675 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117609 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.117675 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117629 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-stats-auth\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.117675 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117643 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.117832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.117832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117708 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3afb35-bfde-45ec-a972-78a6e7461d9d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.117832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117735 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6594a6f-9c78-40bf-99b7-fb21cd031157-config\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.117832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117753 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.117832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117772 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq4g\" (UniqueName: \"kubernetes.io/projected/a547a8cd-8909-475e-a03d-15b409877777-kube-api-access-7nq4g\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.117832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67tn\" (UniqueName: \"kubernetes.io/projected/04efc6e4-19af-4935-a7c1-2dac67498e4c-kube-api-access-q67tn\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.117832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117813 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.118074 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117840 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8743ad27-5d90-4b7f-8012-6faf21548f97-tmp-dir\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.118074 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117861 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.118074 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45q9q\" (UniqueName: \"kubernetes.io/projected/c6eb7759-f63d-4817-9128-adcecdceb0d7-kube-api-access-45q9q\") pod \"volume-data-source-validator-58c9f5f46d-h6cxw\" (UID: \"c6eb7759-f63d-4817-9128-adcecdceb0d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" Oct 30 21:46:08.118074 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.117940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c494bc9-5908-48bc-9825-2669ad4196c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:08.118074 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.118008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.126226 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.126206 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.126226 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.126210 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.128889 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.128874 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:46:08.128889 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.128883 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.129339 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.129305 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:46:08.129486 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.129473 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:46:08.130232 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.130216 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:46:08.135229 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.135215 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:46:08.173066 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173047 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:46:08.173354 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173313 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:46:08.173459 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173372 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:46:08.173459 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173413 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:46:08.173459 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173424 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:46:08.173459 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173414 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:46:08.173459 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173376 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:46:08.173459 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173460 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:46:08.173777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173430 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:46:08.173777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173376 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:46:08.173777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173487 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:46:08.173777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173573 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:46:08.173777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173654 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:46:08.173922 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.173837 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tjx5c\"" Oct 30 21:46:08.174304 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.174292 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:46:08.174391 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.174376 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-879p5\"" Oct 30 21:46:08.174486 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.174411 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:46:08.174486 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.174424 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hs26f\"" Oct 30 21:46:08.175586 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.175572 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g8m85\"" Oct 30 21:46:08.175634 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.175588 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:46:08.176704 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.176685 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qdqzz"] Oct 30 21:46:08.177729 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.177710 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d"] Oct 30 21:46:08.178309 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.178294 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c8c9db44f-llgx4"] Oct 30 21:46:08.178927 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.178910 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fwrp"] Oct 30 21:46:08.179562 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.179546 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr"] Oct 30 21:46:08.179943 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.179926 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7mx\" (UniqueName: \"kubernetes.io/projected/177b7d0a-d868-497a-b648-23228667c743-kube-api-access-7z7mx\") pod \"kas-connection-checker-685895c7c9-dll8d\" (UID: \"177b7d0a-d868-497a-b648-23228667c743\") " pod="kube-system/kas-connection-checker-685895c7c9-dll8d" Oct 30 21:46:08.180021 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.179962 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7xlh\" (UniqueName: \"kubernetes.io/projected/0abc3e92-4ede-4421-ab8d-91477bd6b85e-kube-api-access-r7xlh\") pod \"kas-connection-checker-685895c7c9-p2dr2\" (UID: \"0abc3e92-4ede-4421-ab8d-91477bd6b85e\") " pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" Oct 30 21:46:08.181428 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.181405 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxtvm\" (UniqueName: \"kubernetes.io/projected/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a-kube-api-access-nxtvm\") pod \"kas-connection-checker-685895c7c9-2jxfz\" (UID: \"6ffd4edc-ce40-4a48-b398-c2d907a7cd0a\") " pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" Oct 30 21:46:08.181503 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.181489 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45q9q\" (UniqueName: \"kubernetes.io/projected/c6eb7759-f63d-4817-9128-adcecdceb0d7-kube-api-access-45q9q\") pod \"volume-data-source-validator-58c9f5f46d-h6cxw\" (UID: \"c6eb7759-f63d-4817-9128-adcecdceb0d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" Oct 30 21:46:08.218385 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.218385 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218387 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.218553 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218408 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.218617 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2xf\" (UniqueName: \"kubernetes.io/projected/8743ad27-5d90-4b7f-8012-6faf21548f97-kube-api-access-7s2xf\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.218617 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqd5\" (UniqueName: \"kubernetes.io/projected/d6594a6f-9c78-40bf-99b7-fb21cd031157-kube-api-access-rtqd5\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.218720 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.218773 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218733 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da4fada-55a5-42ad-b873-46f86929c5d0-cert\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.218773 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218749 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.218773 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.218908 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-stats-auth\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.218908 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.218908 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.218908 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.218884 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:08.218908 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.218898 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:08.218908 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3afb35-bfde-45ec-a972-78a6e7461d9d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6594a6f-9c78-40bf-99b7-fb21cd031157-config\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.218951 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.718931667 +0000 UTC m=+31.682536090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218981 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.218992 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq4g\" (UniqueName: \"kubernetes.io/projected/a547a8cd-8909-475e-a03d-15b409877777-kube-api-access-7nq4g\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219031 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67tn\" (UniqueName: \"kubernetes.io/projected/04efc6e4-19af-4935-a7c1-2dac67498e4c-kube-api-access-q67tn\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.219057 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.719039268 +0000 UTC m=+31.682643693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219092 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219121 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8743ad27-5d90-4b7f-8012-6faf21548f97-tmp-dir\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219180 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c494bc9-5908-48bc-9825-2669ad4196c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:08.219205 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219205 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219241 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219308 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a07200b0-a7f6-4c5a-9197-b73779a06c1f-serving-cert\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.219385 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.218989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.219447 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.719424824 +0000 UTC m=+31.683029250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219560 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqdn\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sz9x\" (UniqueName: \"kubernetes.io/projected/2da4fada-55a5-42ad-b873-46f86929c5d0-kube-api-access-5sz9x\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219619 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6594a6f-9c78-40bf-99b7-fb21cd031157-serving-cert\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219645 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvtf\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.219811 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219652 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8743ad27-5d90-4b7f-8012-6faf21548f97-tmp-dir\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3afb35-bfde-45ec-a972-78a6e7461d9d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219673 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3afb35-bfde-45ec-a972-78a6e7461d9d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219697 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04efc6e4-19af-4935-a7c1-2dac67498e4c-serving-cert\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwfx\" (UniqueName: \"kubernetes.io/projected/4f3afb35-bfde-45ec-a972-78a6e7461d9d-kube-api-access-thwfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.219791 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.219831 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.719819075 +0000 UTC m=+31.683423523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-snapshots\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcqn\" (UniqueName: \"kubernetes.io/projected/a07200b0-a7f6-4c5a-9197-b73779a06c1f-kube-api-access-rpcqn\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219936 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219988 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-default-certificate\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.219996 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqg42\" (UniqueName: \"kubernetes.io/projected/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-kube-api-access-cqg42\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220051 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6594a6f-9c78-40bf-99b7-fb21cd031157-config\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.220615 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220055 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-tmp\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8743ad27-5d90-4b7f-8012-6faf21548f97-config-volume\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220176 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd5v\" (UniqueName: \"kubernetes.io/projected/7e37f363-3bfe-4f18-a5ee-f60f849499d1-kube-api-access-7bd5v\") pod \"network-check-source-6f78d59f99-qzvdr\" (UID: \"7e37f363-3bfe-4f18-a5ee-f60f849499d1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxh4\" (UniqueName: \"kubernetes.io/projected/6b4c949e-cff6-4abf-b3ff-f85046f76784-kube-api-access-2pxh4\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220232 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220258 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220293 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-config\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220339 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.220434 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.220473 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.720460954 +0000 UTC m=+31.684065368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220506 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220506 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-tmp\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.220591 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:08.221201 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.220628 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.720615365 +0000 UTC m=+31.684219795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:08.221878 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.220713 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.222463 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.222380 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.223099 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.223074 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da4fada-55a5-42ad-b873-46f86929c5d0-cert\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.223192 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.223103 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8743ad27-5d90-4b7f-8012-6faf21548f97-config-volume\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.223192 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.223142 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-snapshots\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.223397 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.223375 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.223478 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.223434 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04efc6e4-19af-4935-a7c1-2dac67498e4c-serving-cert\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.224688 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.223811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.224688 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.223867 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.224688 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.223920 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.72390317 +0000 UTC m=+31.687507604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : configmap references non-existent config key: service-ca.crt Oct 30 21:46:08.226603 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.225434 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-stats-auth\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.226603 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.225837 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:08.226603 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.225852 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:08.226603 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.225904 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.725890908 +0000 UTC m=+31.689495326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:08.226603 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.226546 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-config\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.228884 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.227265 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3afb35-bfde-45ec-a972-78a6e7461d9d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.228884 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.227343 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c494bc9-5908-48bc-9825-2669ad4196c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:08.228884 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.228693 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.229079 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.229058 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-default-certificate\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.231585 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.231562 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.231674 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.231572 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6594a6f-9c78-40bf-99b7-fb21cd031157-serving-cert\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.233237 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.233204 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.233445 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.233423 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.233494 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.233446 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.234059 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.234034 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a07200b0-a7f6-4c5a-9197-b73779a06c1f-serving-cert\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.240414 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.240384 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" Oct 30 21:46:08.246665 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.246646 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dll8d" Oct 30 21:46:08.252375 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.252360 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" Oct 30 21:46:08.257937 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.257921 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" Oct 30 21:46:08.285038 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.285010 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.286138 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.286104 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq4g\" (UniqueName: \"kubernetes.io/projected/a547a8cd-8909-475e-a03d-15b409877777-kube-api-access-7nq4g\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.286578 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.286542 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2xf\" (UniqueName: \"kubernetes.io/projected/8743ad27-5d90-4b7f-8012-6faf21548f97-kube-api-access-7s2xf\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.287586 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.287561 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcqn\" (UniqueName: \"kubernetes.io/projected/a07200b0-a7f6-4c5a-9197-b73779a06c1f-kube-api-access-rpcqn\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.288065 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.288043 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqg42\" (UniqueName: \"kubernetes.io/projected/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-kube-api-access-cqg42\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.290121 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.290092 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvtf\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.292617 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.292592 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxh4\" (UniqueName: \"kubernetes.io/projected/6b4c949e-cff6-4abf-b3ff-f85046f76784-kube-api-access-2pxh4\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.293392 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.293373 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sz9x\" (UniqueName: \"kubernetes.io/projected/2da4fada-55a5-42ad-b873-46f86929c5d0-kube-api-access-5sz9x\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.293788 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.293770 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqdn\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.298414 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.297674 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd5v\" (UniqueName: \"kubernetes.io/projected/7e37f363-3bfe-4f18-a5ee-f60f849499d1-kube-api-access-7bd5v\") pod \"network-check-source-6f78d59f99-qzvdr\" (UID: \"7e37f363-3bfe-4f18-a5ee-f60f849499d1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" Oct 30 21:46:08.298414 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.297715 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwfx\" (UniqueName: \"kubernetes.io/projected/4f3afb35-bfde-45ec-a972-78a6e7461d9d-kube-api-access-thwfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.301825 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.299867 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqd5\" (UniqueName: \"kubernetes.io/projected/d6594a6f-9c78-40bf-99b7-fb21cd031157-kube-api-access-rtqd5\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.301825 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.300368 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67tn\" (UniqueName: \"kubernetes.io/projected/04efc6e4-19af-4935-a7c1-2dac67498e4c-kube-api-access-q67tn\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.302116 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.302097 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.336133 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.336102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:08.368956 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.368932 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" Oct 30 21:46:08.375438 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.375412 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:08.521398 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.521367 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:08.521398 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.521387 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:08.564536 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.564514 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:08.571577 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.571553 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:46:08.571577 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.571553 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xlfcr\"" Oct 30 21:46:08.571713 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.571584 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\"" Oct 30 21:46:08.571713 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.571676 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:46:08.575768 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.575736 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:08.586664 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.586647 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:08.643313 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.643030 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw"] Oct 30 21:46:08.646598 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.646579 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p2dr2"] Oct 30 21:46:08.656546 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.653589 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dll8d"] Oct 30 21:46:08.657095 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.657037 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eb7759_f63d_4817_9128_adcecdceb0d7.slice/crio-e050a36b4e73089b079cde1c083211d3ce46e7d94b274acf395c651303e3afec WatchSource:0}: Error finding container e050a36b4e73089b079cde1c083211d3ce46e7d94b274acf395c651303e3afec: Status 404 returned error can't find the container with id e050a36b4e73089b079cde1c083211d3ce46e7d94b274acf395c651303e3afec Oct 30 21:46:08.663435 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.663414 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abc3e92_4ede_4421_ab8d_91477bd6b85e.slice/crio-ce78d7cc07629e2740d42c541a2bdf9eb56b89633de63ae123ea6b26d4ea8a3e WatchSource:0}: Error finding container ce78d7cc07629e2740d42c541a2bdf9eb56b89633de63ae123ea6b26d4ea8a3e: Status 404 returned error can't find the container with id ce78d7cc07629e2740d42c541a2bdf9eb56b89633de63ae123ea6b26d4ea8a3e Oct 30 21:46:08.665149 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.665117 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177b7d0a_d868_497a_b648_23228667c743.slice/crio-f7bb17a940c0a395c77520a0b2daea86209f8b4890de812cbf53ce7d4581f639 WatchSource:0}: Error finding container f7bb17a940c0a395c77520a0b2daea86209f8b4890de812cbf53ce7d4581f639: Status 404 returned error can't find the container with id f7bb17a940c0a395c77520a0b2daea86209f8b4890de812cbf53ce7d4581f639 Oct 30 21:46:08.706605 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.706575 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2jxfz"] Oct 30 21:46:08.716143 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.715473 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffd4edc_ce40_4a48_b398_c2d907a7cd0a.slice/crio-8b8750494f7b96e46fe5038c681ed8e500e6db93da09b8784b689bc296fd5efb WatchSource:0}: Error finding container 8b8750494f7b96e46fe5038c681ed8e500e6db93da09b8784b689bc296fd5efb: Status 404 returned error can't find the container with id 8b8750494f7b96e46fe5038c681ed8e500e6db93da09b8784b689bc296fd5efb Oct 30 21:46:08.719978 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.718559 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dll8d" event={"ID":"177b7d0a-d868-497a-b648-23228667c743","Type":"ContainerStarted","Data":"f7bb17a940c0a395c77520a0b2daea86209f8b4890de812cbf53ce7d4581f639"} Oct 30 21:46:08.721168 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.721084 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" event={"ID":"0abc3e92-4ede-4421-ab8d-91477bd6b85e","Type":"ContainerStarted","Data":"ce78d7cc07629e2740d42c541a2bdf9eb56b89633de63ae123ea6b26d4ea8a3e"} Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.723774 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" event={"ID":"c6eb7759-f63d-4817-9128-adcecdceb0d7","Type":"ContainerStarted","Data":"e050a36b4e73089b079cde1c083211d3ce46e7d94b274acf395c651303e3afec"} Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.724340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.724377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.724405 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.724454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724459 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724478 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.724493 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724528 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.724511103 +0000 UTC m=+32.688115526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.724577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724599 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.724579973 +0000 UTC m=+32.688184406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : configmap references non-existent config key: service-ca.crt Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724640 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724671 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.724661628 +0000 UTC m=+32.688266066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.724687 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:08.725083 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724811 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:08.725890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724842 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.724832214 +0000 UTC m=+32.688436668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:08.725890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724899 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:08.725890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724927 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.724918181 +0000 UTC m=+32.688522595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:08.725890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.724973 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:08.725890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.725013 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.724990524 +0000 UTC m=+32.688594977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:08.725890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.725012 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:08.725890 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.725054 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.725041591 +0000 UTC m=+32.688646002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:08.761353 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.760480 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr"] Oct 30 21:46:08.761453 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.761426 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qdqzz"] Oct 30 21:46:08.762763 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.762746 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tc7w4"] Oct 30 21:46:08.772877 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.772517 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da4fada_55a5_42ad_b873_46f86929c5d0.slice/crio-3ee752b50093429980c3751f2b20b18a96ee782759cac4150a6654a1f4a5ffb3 WatchSource:0}: Error finding container 3ee752b50093429980c3751f2b20b18a96ee782759cac4150a6654a1f4a5ffb3: Status 404 returned error can't find the container with id 3ee752b50093429980c3751f2b20b18a96ee782759cac4150a6654a1f4a5ffb3 Oct 30 21:46:08.775415 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.774506 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07200b0_a7f6_4c5a_9197_b73779a06c1f.slice/crio-49b2fd0dd2fd5d1086abb56f810fb359086a6ba1020a275a75c10d3381ff6cc2 WatchSource:0}: Error finding container 49b2fd0dd2fd5d1086abb56f810fb359086a6ba1020a275a75c10d3381ff6cc2: Status 404 returned error can't find the container with id 49b2fd0dd2fd5d1086abb56f810fb359086a6ba1020a275a75c10d3381ff6cc2 Oct 30 21:46:08.775415 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.774975 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e37f363_3bfe_4f18_a5ee_f60f849499d1.slice/crio-1445753add3bfd08ac88d593db19e31333214b4625d83a4cb70701fc4d2f74b3 WatchSource:0}: Error finding container 1445753add3bfd08ac88d593db19e31333214b4625d83a4cb70701fc4d2f74b3: Status 404 returned error can't find the container with id 1445753add3bfd08ac88d593db19e31333214b4625d83a4cb70701fc4d2f74b3 Oct 30 21:46:08.825464 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.825441 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:08.825748 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.825727 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:08.825748 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.825749 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:08.825877 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:08.825800 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:09.825784234 +0000 UTC m=+32.789388646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:08.947041 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.947012 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p"] Oct 30 21:46:08.953272 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.953250 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6594a6f_9c78_40bf_99b7_fb21cd031157.slice/crio-4bbafb7abf8ec0f03f073c5f55e107eb00f23c3270c1bbdcaed1182a2a28ce99 WatchSource:0}: Error finding container 4bbafb7abf8ec0f03f073c5f55e107eb00f23c3270c1bbdcaed1182a2a28ce99: Status 404 returned error can't find the container with id 4bbafb7abf8ec0f03f073c5f55e107eb00f23c3270c1bbdcaed1182a2a28ce99 Oct 30 21:46:08.981908 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.981888 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c"] Oct 30 21:46:08.985791 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:08.985771 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jqbm4"] Oct 30 21:46:08.988378 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.988356 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3afb35_bfde_45ec_a972_78a6e7461d9d.slice/crio-b6b8d110b3d8a43cb1de351c753c9e78f1a1071294c70f1f61d2ed8ef4f14400 WatchSource:0}: Error finding container b6b8d110b3d8a43cb1de351c753c9e78f1a1071294c70f1f61d2ed8ef4f14400: Status 404 returned error can't find the container with id b6b8d110b3d8a43cb1de351c753c9e78f1a1071294c70f1f61d2ed8ef4f14400 Oct 30 21:46:08.991744 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:08.991719 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04efc6e4_19af_4935_a7c1_2dac67498e4c.slice/crio-5264bc20d60def2a0748f92e789e9ccaa9102a8306204f3ac4f48631d926c53c WatchSource:0}: Error finding container 5264bc20d60def2a0748f92e789e9ccaa9102a8306204f3ac4f48631d926c53c: Status 404 returned error can't find the container with id 5264bc20d60def2a0748f92e789e9ccaa9102a8306204f3ac4f48631d926c53c Oct 30 21:46:09.522061 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.522033 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:09.571806 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.571786 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5wdm\"" Oct 30 21:46:09.727041 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.727009 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc7w4" event={"ID":"a07200b0-a7f6-4c5a-9197-b73779a06c1f","Type":"ContainerStarted","Data":"49b2fd0dd2fd5d1086abb56f810fb359086a6ba1020a275a75c10d3381ff6cc2"} Oct 30 21:46:09.728102 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.728077 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" event={"ID":"4f3afb35-bfde-45ec-a972-78a6e7461d9d","Type":"ContainerStarted","Data":"b6b8d110b3d8a43cb1de351c753c9e78f1a1071294c70f1f61d2ed8ef4f14400"} Oct 30 21:46:09.729361 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.729335 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" event={"ID":"6ffd4edc-ce40-4a48-b398-c2d907a7cd0a","Type":"ContainerStarted","Data":"67bb04393d49f3e711b57a9c5dd48013b0b6f9cf5b46f191fe1a287e6fafa85a"} Oct 30 21:46:09.729463 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.729376 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" event={"ID":"6ffd4edc-ce40-4a48-b398-c2d907a7cd0a","Type":"ContainerStarted","Data":"8b8750494f7b96e46fe5038c681ed8e500e6db93da09b8784b689bc296fd5efb"} Oct 30 21:46:09.730369 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.730343 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" event={"ID":"d6594a6f-9c78-40bf-99b7-fb21cd031157","Type":"ContainerStarted","Data":"4bbafb7abf8ec0f03f073c5f55e107eb00f23c3270c1bbdcaed1182a2a28ce99"} Oct 30 21:46:09.731442 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.731422 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" event={"ID":"7e37f363-3bfe-4f18-a5ee-f60f849499d1","Type":"ContainerStarted","Data":"1445753add3bfd08ac88d593db19e31333214b4625d83a4cb70701fc4d2f74b3"} Oct 30 21:46:09.732755 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.732739 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dll8d" event={"ID":"177b7d0a-d868-497a-b648-23228667c743","Type":"ContainerStarted","Data":"8235a5ae54d57c445df4668b224059289d2f974c36f76122230e099deb0d7fff"} Oct 30 21:46:09.733812 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.733793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:09.733897 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.733819 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:09.733897 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.733841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:09.733995 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.733931 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:09.733995 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.733937 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:09.733995 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.733951 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:09.733995 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.733968 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.733957164 +0000 UTC m=+34.697561575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:09.733995 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.733968 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" event={"ID":"0abc3e92-4ede-4421-ab8d-91477bd6b85e","Type":"ContainerStarted","Data":"fc5262f42362019dcbf321122d044f1b7a83e8c8dde8bb308eddd2ca3aa0ed1a"} Oct 30 21:46:09.733995 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.733996 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.73397929 +0000 UTC m=+34.697583730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734034 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.734060 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.734104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734122 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.734106911 +0000 UTC m=+34.697711327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.734188 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734213 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.734257 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734305 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.734291208 +0000 UTC m=+34.697895652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : configmap references non-existent config key: service-ca.crt Oct 30 21:46:09.734333 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734333 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:09.734659 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734342 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.734314415 +0000 UTC m=+34.697918829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:09.734659 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734350 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:09.734659 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734375 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.73436492 +0000 UTC m=+34.697969331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:09.734659 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.734412 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.734392216 +0000 UTC m=+34.697996630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:09.735178 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.735159 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" event={"ID":"04efc6e4-19af-4935-a7c1-2dac67498e4c","Type":"ContainerStarted","Data":"5264bc20d60def2a0748f92e789e9ccaa9102a8306204f3ac4f48631d926c53c"} Oct 30 21:46:09.736170 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.736150 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qdqzz" event={"ID":"2da4fada-55a5-42ad-b873-46f86929c5d0","Type":"ContainerStarted","Data":"3ee752b50093429980c3751f2b20b18a96ee782759cac4150a6654a1f4a5ffb3"} Oct 30 21:46:09.835143 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:09.835088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:09.835256 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.835182 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:09.835256 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.835193 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:09.835256 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:09.835234 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:11.835219234 +0000 UTC m=+34.798823659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:10.033488 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:10.033429 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" podStartSLOduration=409.03341197 podStartE2EDuration="6m49.03341197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:10.032153221 +0000 UTC m=+32.995757654" watchObservedRunningTime="2026-10-30 21:46:10.03341197 +0000 UTC m=+32.997016405" Oct 30 21:46:10.264413 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:10.264341 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dll8d" podStartSLOduration=409.264308884 podStartE2EDuration="6m49.264308884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:10.263482753 +0000 UTC m=+33.227087184" watchObservedRunningTime="2026-10-30 21:46:10.264308884 +0000 UTC m=+33.227913318" Oct 30 21:46:10.264609 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:10.264455 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" podStartSLOduration=409.264450468 podStartE2EDuration="6m49.264450468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:10.153558924 +0000 UTC m=+33.117163362" watchObservedRunningTime="2026-10-30 21:46:10.264450468 +0000 UTC m=+33.228054904" Oct 30 21:46:11.353611 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.353387 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:11.353611 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.353452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:11.366860 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.362649 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:46:11.366860 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.362731 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:43.362711785 +0000 UTC m=+66.326316203 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : secret "metrics-daemon-secret" not found Oct 30 21:46:11.368876 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.368854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:11.463351 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.463009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:11.480274 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.480026 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:11.587914 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.587724 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:11.660998 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.660852 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:11.749536 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.749508 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" event={"ID":"c6eb7759-f63d-4817-9128-adcecdceb0d7","Type":"ContainerStarted","Data":"fbc6f663e0a364fa321af275e49cb50ac6559e05fadfd09f47ec02e13c1bf79a"} Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.767387 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.767584 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.767706 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.767686908 +0000 UTC m=+38.731291336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.768059 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.768143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.768180 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.768221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:11.768983 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.768283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:11.770029 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.770008 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:11.770840 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.770485 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:11.770840 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.770560 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:11.770840 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.770695 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.771048 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771129 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.77110939 +0000 UTC m=+38.734713808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : configmap references non-existent config key: service-ca.crt Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771151 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.771140593 +0000 UTC m=+38.734745012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771168 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.771158308 +0000 UTC m=+38.734762723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771169 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771182 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771182 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.771173735 +0000 UTC m=+38.734778147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:11.771218 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771199 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.771191715 +0000 UTC m=+38.734796193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:11.771660 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.771232 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.771221683 +0000 UTC m=+38.734826107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:11.862415 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.862294 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" podStartSLOduration=380.060785078 podStartE2EDuration="6m22.862275776s" totalImagesPullingTime="2.801490698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:11.861359019 +0000 UTC m=+34.824963451" watchObservedRunningTime="2026-10-30 21:46:11.862275776 +0000 UTC m=+34.825880211" Oct 30 21:46:11.871791 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.871764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:11.871914 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.871900 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:11.871948 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.871914 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:11.872057 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:11.872019 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.872004114 +0000 UTC m=+38.835608547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:11.985777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:11.985745 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zhvv"] Oct 30 21:46:11.992497 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:11.992458 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3280f18_2d95_4303_8f34_5a2fe91671e6.slice/crio-3ef0864688ddab0b08ffbaefe76d5089f9544f86748c885c9e4a73596a7370c6 WatchSource:0}: Error finding container 3ef0864688ddab0b08ffbaefe76d5089f9544f86748c885c9e4a73596a7370c6: Status 404 returned error can't find the container with id 3ef0864688ddab0b08ffbaefe76d5089f9544f86748c885c9e4a73596a7370c6 Oct 30 21:46:12.098586 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:12.098530 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqcc7"] Oct 30 21:46:12.115635 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:12.115558 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573e461a_8123_4e53_9134_80d6eb9a3e71.slice/crio-6cf1f65e9e7d6f4ecabbf5decae65ab7f71a07bc289c1619c58a4e58116c233b WatchSource:0}: Error finding container 6cf1f65e9e7d6f4ecabbf5decae65ab7f71a07bc289c1619c58a4e58116c233b: Status 404 returned error can't find the container with id 6cf1f65e9e7d6f4ecabbf5decae65ab7f71a07bc289c1619c58a4e58116c233b Oct 30 21:46:12.755229 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:12.755185 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqcc7" event={"ID":"573e461a-8123-4e53-9134-80d6eb9a3e71","Type":"ContainerStarted","Data":"6cf1f65e9e7d6f4ecabbf5decae65ab7f71a07bc289c1619c58a4e58116c233b"} Oct 30 21:46:12.757792 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:12.757738 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4zhvv" event={"ID":"d3280f18-2d95-4303-8f34-5a2fe91671e6","Type":"ContainerStarted","Data":"3ef0864688ddab0b08ffbaefe76d5089f9544f86748c885c9e4a73596a7370c6"} Oct 30 21:46:14.772114 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.772072 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" event={"ID":"d6594a6f-9c78-40bf-99b7-fb21cd031157","Type":"ContainerStarted","Data":"b59f77cc7ce0ff51bc5082a98e457a970cdef300c154434959dcd19a769ca6f5"} Oct 30 21:46:14.777058 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.777036 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/0.log" Oct 30 21:46:14.777181 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.777067 2571 generic.go:356] "Generic (PLEG): container finished" podID="04efc6e4-19af-4935-a7c1-2dac67498e4c" containerID="3885bb1cf018e4bff5700b27e38ea3bd444db5fb7aa898030c7ef48571fc5f7e" exitCode=255 Oct 30 21:46:14.777181 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.777135 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" event={"ID":"04efc6e4-19af-4935-a7c1-2dac67498e4c","Type":"ContainerDied","Data":"3885bb1cf018e4bff5700b27e38ea3bd444db5fb7aa898030c7ef48571fc5f7e"} Oct 30 21:46:14.777495 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.777477 2571 scope.go:122] "RemoveContainer" containerID="3885bb1cf018e4bff5700b27e38ea3bd444db5fb7aa898030c7ef48571fc5f7e" Oct 30 21:46:14.780662 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.779920 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc7w4" event={"ID":"a07200b0-a7f6-4c5a-9197-b73779a06c1f","Type":"ContainerStarted","Data":"2cfc479fb7215b646116e099071808a6ca897c5b3fa4c932c37b65f16df57f28"} Oct 30 21:46:14.786832 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.785992 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" event={"ID":"4f3afb35-bfde-45ec-a972-78a6e7461d9d","Type":"ContainerStarted","Data":"0318d79587cf816f7900abf9e720c0e26aa372c17706afec28febb209bb4111d"} Oct 30 21:46:14.900543 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.899463 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bhs7l"] Oct 30 21:46:14.909944 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.909926 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:14.923925 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.922843 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" podStartSLOduration=380.415254689 podStartE2EDuration="6m25.922825671s" totalImagesPullingTime="5.507570982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:14.922312108 +0000 UTC m=+37.885916546" watchObservedRunningTime="2026-10-30 21:46:14.922825671 +0000 UTC m=+37.886430106" Oct 30 21:46:14.963570 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.963492 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhs7l"] Oct 30 21:46:14.969475 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.969147 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:46:14.969475 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.969330 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:46:14.969475 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:14.969429 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pz9m9\"" Oct 30 21:46:15.004387 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.003877 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnkh\" (UniqueName: \"kubernetes.io/projected/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-api-access-2fnkh\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.004387 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.003918 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e021343-49b5-4868-a1f8-3a1bd283c42c-crio-socket\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.004387 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.003957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.004387 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.004027 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e021343-49b5-4868-a1f8-3a1bd283c42c-data-volume\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.004387 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.004080 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.105436 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.105355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.105582 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.105464 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e021343-49b5-4868-a1f8-3a1bd283c42c-data-volume\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.105582 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.105471 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:46:15.105582 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.105519 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.105582 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.105538 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.605518849 +0000 UTC m=+38.569123301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : configmap "kube-rbac-proxy" not found Oct 30 21:46:15.105792 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.105609 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:15.105792 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.105666 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:15.605651312 +0000 UTC m=+38.569255726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:15.105792 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.105696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnkh\" (UniqueName: \"kubernetes.io/projected/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-api-access-2fnkh\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.105792 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.105722 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e021343-49b5-4868-a1f8-3a1bd283c42c-crio-socket\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.105977 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.105928 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e021343-49b5-4868-a1f8-3a1bd283c42c-crio-socket\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.106274 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.106243 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e021343-49b5-4868-a1f8-3a1bd283c42c-data-volume\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.170214 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.170186 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnkh\" (UniqueName: \"kubernetes.io/projected/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-api-access-2fnkh\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.175561 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.175534 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-p8p6k-4z6rv"] Oct 30 21:46:15.192745 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.192717 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.206653 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.206624 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.206782 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.206728 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.206889 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.206802 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdh8\" (UniqueName: \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.206954 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.206890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.236152 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.236129 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p8p6k-4z6rv"] Oct 30 21:46:15.247937 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.247916 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kmf25\"" Oct 30 21:46:15.272638 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.272584 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tc7w4" podStartSLOduration=382.361100001 podStartE2EDuration="6m27.272567284s" totalImagesPullingTime="4.911467283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:15.271987959 +0000 UTC m=+38.235592391" watchObservedRunningTime="2026-10-30 21:46:15.272567284 +0000 UTC m=+38.236171718" Oct 30 21:46:15.309748 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.308086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdh8\" (UniqueName: \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.309748 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.308183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.309748 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.308254 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.309748 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.308343 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.309748 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.308737 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.309748 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.309702 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.310104 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.309790 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.374354 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.374248 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdh8\" (UniqueName: \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.508545 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.508490 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" podStartSLOduration=381.050759784 podStartE2EDuration="6m26.508471698s" totalImagesPullingTime="5.457711914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:15.396558472 +0000 UTC m=+38.360162905" watchObservedRunningTime="2026-10-30 21:46:15.508471698 +0000 UTC m=+38.472076132" Oct 30 21:46:15.528063 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.528041 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:15.611032 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.610479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.611032 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.610610 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:15.611032 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.610853 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:15.611032 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.610912 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:16.610894462 +0000 UTC m=+39.574498872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:15.611032 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.610962 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:46:15.611032 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.610993 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:16.610983631 +0000 UTC m=+39.574588046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : configmap "kube-rbac-proxy" not found Oct 30 21:46:15.794807 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.791815 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:46:15.794807 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.792357 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/0.log" Oct 30 21:46:15.794807 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.792389 2571 generic.go:356] "Generic (PLEG): container finished" podID="04efc6e4-19af-4935-a7c1-2dac67498e4c" containerID="c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b" exitCode=255 Oct 30 21:46:15.794807 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.792456 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" event={"ID":"04efc6e4-19af-4935-a7c1-2dac67498e4c","Type":"ContainerDied","Data":"c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b"} Oct 30 21:46:15.794807 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.792501 2571 scope.go:122] "RemoveContainer" containerID="3885bb1cf018e4bff5700b27e38ea3bd444db5fb7aa898030c7ef48571fc5f7e" Oct 30 21:46:15.794807 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.792863 2571 scope.go:122] "RemoveContainer" containerID="c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b" Oct 30 21:46:15.794807 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.793087 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jqbm4_openshift-console-operator(04efc6e4-19af-4935-a7c1-2dac67498e4c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" podUID="04efc6e4-19af-4935-a7c1-2dac67498e4c" Oct 30 21:46:15.796548 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.796521 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qdqzz" event={"ID":"2da4fada-55a5-42ad-b873-46f86929c5d0","Type":"ContainerStarted","Data":"d997ded3c128a30191142fc20fd28c6a4499210b12cda4180b114dc764644562"} Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.812745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.812817 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.812877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.812942 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.812996 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.812977235 +0000 UTC m=+46.776581657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.813081 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.813192 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.813258 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:15.813343 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.813284 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:15.813869 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.813615 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.81359764 +0000 UTC m=+46.777202055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : configmap references non-existent config key: service-ca.crt Oct 30 21:46:15.813869 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.813809 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:15.813869 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.813847 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.81383398 +0000 UTC m=+46.777438390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:15.814131 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814115 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:15.814196 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814152 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.814141665 +0000 UTC m=+46.777746119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:15.814583 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814557 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:15.814674 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814611 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.814596677 +0000 UTC m=+46.778201097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:15.814674 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814658 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:15.814674 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814674 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:15.814818 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814716 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.814701774 +0000 UTC m=+46.778306203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:15.814818 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814659 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:15.814818 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.814761 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.81475162 +0000 UTC m=+46.778356069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:15.914183 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.914159 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:15.914339 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.914304 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:15.914339 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.914335 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:15.914448 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:15.914388 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:23.914367426 +0000 UTC m=+46.877971838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:15.982686 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:15.982657 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p8p6k-4z6rv"] Oct 30 21:46:15.990454 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:15.990429 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd83615_d417_4d88_88d2_9f74ffbce102.slice/crio-5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4 WatchSource:0}: Error finding container 5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4: Status 404 returned error can't find the container with id 5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4 Oct 30 21:46:16.043050 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.043007 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qdqzz" podStartSLOduration=2.399033173 podStartE2EDuration="9.042992754s" totalImagesPullingTime="6.643959581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:16.042403992 +0000 UTC m=+39.006008408" watchObservedRunningTime="2026-10-30 21:46:16.042992754 +0000 UTC m=+39.006597186" Oct 30 21:46:16.566074 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.566040 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5"] Oct 30 21:46:16.581941 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.581910 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" Oct 30 21:46:16.583241 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.583216 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb"] Oct 30 21:46:16.595980 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.595956 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" Oct 30 21:46:16.619796 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.619774 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:16.619898 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.619876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxspb\" (UniqueName: \"kubernetes.io/projected/3ac87e61-9d62-4657-90a3-2bf7e4877f85-kube-api-access-fxspb\") pod \"migrator-66df5d496b-5vnv5\" (UID: \"3ac87e61-9d62-4657-90a3-2bf7e4877f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" Oct 30 21:46:16.619981 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.619960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl44l\" (UniqueName: \"kubernetes.io/projected/74d26ea6-81b0-4ff7-acef-cd28751f4a78-kube-api-access-sl44l\") pod \"migrator-66df5d496b-ssjmb\" (UID: \"74d26ea6-81b0-4ff7-acef-cd28751f4a78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" Oct 30 21:46:16.620088 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:16.619986 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:16.620088 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.619998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:16.620088 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:16.620051 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:18.620038867 +0000 UTC m=+41.583643298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:16.620088 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:16.620063 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:46:16.620236 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:16.620223 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:18.620210266 +0000 UTC m=+41.583814681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : configmap "kube-rbac-proxy" not found Oct 30 21:46:16.631311 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.631284 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5"] Oct 30 21:46:16.632508 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.632462 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tmn85\"" Oct 30 21:46:16.633855 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.633785 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:16.633969 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.633857 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:46:16.646695 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.646672 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb"] Oct 30 21:46:16.720745 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.720700 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxspb\" (UniqueName: \"kubernetes.io/projected/3ac87e61-9d62-4657-90a3-2bf7e4877f85-kube-api-access-fxspb\") pod \"migrator-66df5d496b-5vnv5\" (UID: \"3ac87e61-9d62-4657-90a3-2bf7e4877f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" Oct 30 21:46:16.720947 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.720785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl44l\" (UniqueName: \"kubernetes.io/projected/74d26ea6-81b0-4ff7-acef-cd28751f4a78-kube-api-access-sl44l\") pod \"migrator-66df5d496b-ssjmb\" (UID: \"74d26ea6-81b0-4ff7-acef-cd28751f4a78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" Oct 30 21:46:16.781274 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.781245 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxspb\" (UniqueName: \"kubernetes.io/projected/3ac87e61-9d62-4657-90a3-2bf7e4877f85-kube-api-access-fxspb\") pod \"migrator-66df5d496b-5vnv5\" (UID: \"3ac87e61-9d62-4657-90a3-2bf7e4877f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" Oct 30 21:46:16.781404 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.781364 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl44l\" (UniqueName: \"kubernetes.io/projected/74d26ea6-81b0-4ff7-acef-cd28751f4a78-kube-api-access-sl44l\") pod \"migrator-66df5d496b-ssjmb\" (UID: \"74d26ea6-81b0-4ff7-acef-cd28751f4a78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" Oct 30 21:46:16.801294 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.801274 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:46:16.801843 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.801597 2571 scope.go:122] "RemoveContainer" containerID="c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b" Oct 30 21:46:16.801843 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:16.801768 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jqbm4_openshift-console-operator(04efc6e4-19af-4935-a7c1-2dac67498e4c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" podUID="04efc6e4-19af-4935-a7c1-2dac67498e4c" Oct 30 21:46:16.803419 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.803343 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" event={"ID":"fbd83615-d417-4d88-88d2-9f74ffbce102","Type":"ContainerStarted","Data":"553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510"} Oct 30 21:46:16.803419 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.803394 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" event={"ID":"fbd83615-d417-4d88-88d2-9f74ffbce102","Type":"ContainerStarted","Data":"5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4"} Oct 30 21:46:16.804972 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.804946 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" event={"ID":"7e37f363-3bfe-4f18-a5ee-f60f849499d1","Type":"ContainerStarted","Data":"6cedd660729094d1d49c10d8d66ae112e438180b0b790098cc5f77e32461952d"} Oct 30 21:46:16.807295 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.807238 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqcc7" event={"ID":"573e461a-8123-4e53-9134-80d6eb9a3e71","Type":"ContainerStarted","Data":"33dde250e04e082b495b5a3e4d9fc8997e20cc6293640ac5ca275922677e4b4c"} Oct 30 21:46:16.807465 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.807448 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:16.910983 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.910910 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" Oct 30 21:46:16.928965 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:16.928929 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" Oct 30 21:46:17.038718 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.037001 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" podStartSLOduration=2.03698272 podStartE2EDuration="2.03698272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:17.034637246 +0000 UTC m=+39.998241680" watchObservedRunningTime="2026-10-30 21:46:17.03698272 +0000 UTC m=+40.000587154" Oct 30 21:46:17.148340 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.148260 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" podStartSLOduration=380.078279304 podStartE2EDuration="6m27.148245706s" totalImagesPullingTime="7.069966402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:17.146708532 +0000 UTC m=+40.110312969" watchObservedRunningTime="2026-10-30 21:46:17.148245706 +0000 UTC m=+40.111850138" Oct 30 21:46:17.259829 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.259782 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tqcc7" podStartSLOduration=35.213329389 podStartE2EDuration="39.259768316s" totalImagesPullingTime="4.046438927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:17.2579928 +0000 UTC m=+40.221597227" watchObservedRunningTime="2026-10-30 21:46:17.259768316 +0000 UTC m=+40.223372750" Oct 30 21:46:17.313161 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.313130 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5"] Oct 30 21:46:17.319908 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:17.319879 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac87e61_9d62_4657_90a3_2bf7e4877f85.slice/crio-7c7bf08e8efb8670722e2a9d6381dd17cbb9d77bc7126502c92f5a45c302ee15 WatchSource:0}: Error finding container 7c7bf08e8efb8670722e2a9d6381dd17cbb9d77bc7126502c92f5a45c302ee15: Status 404 returned error can't find the container with id 7c7bf08e8efb8670722e2a9d6381dd17cbb9d77bc7126502c92f5a45c302ee15 Oct 30 21:46:17.367281 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.367252 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb"] Oct 30 21:46:17.372936 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:17.372910 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d26ea6_81b0_4ff7_acef_cd28751f4a78.slice/crio-7d13a38f48e0d71f1c0565f93427760ce25fa590e82ed05cfac225dbbc543ae3 WatchSource:0}: Error finding container 7d13a38f48e0d71f1c0565f93427760ce25fa590e82ed05cfac225dbbc543ae3: Status 404 returned error can't find the container with id 7d13a38f48e0d71f1c0565f93427760ce25fa590e82ed05cfac225dbbc543ae3 Oct 30 21:46:17.597493 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.597403 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7hrj8"] Oct 30 21:46:17.628836 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.628809 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.655755 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.655729 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7hrj8"] Oct 30 21:46:17.678604 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.678578 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:46:17.678604 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.678580 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:46:17.678777 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.678738 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:46:17.678826 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.678805 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5mqqn\"" Oct 30 21:46:17.679261 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.679245 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:46:17.729888 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.729834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f793981d-e86b-4866-920e-aa96d4de9e30-signing-cabundle\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.729997 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.729972 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f793981d-e86b-4866-920e-aa96d4de9e30-signing-key\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.730062 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.730044 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95nz\" (UniqueName: \"kubernetes.io/projected/f793981d-e86b-4866-920e-aa96d4de9e30-kube-api-access-k95nz\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.812862 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.812827 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" event={"ID":"74d26ea6-81b0-4ff7-acef-cd28751f4a78","Type":"ContainerStarted","Data":"7d13a38f48e0d71f1c0565f93427760ce25fa590e82ed05cfac225dbbc543ae3"} Oct 30 21:46:17.814126 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.814102 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" event={"ID":"3ac87e61-9d62-4657-90a3-2bf7e4877f85","Type":"ContainerStarted","Data":"7c7bf08e8efb8670722e2a9d6381dd17cbb9d77bc7126502c92f5a45c302ee15"} Oct 30 21:46:17.831172 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.831147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f793981d-e86b-4866-920e-aa96d4de9e30-signing-cabundle\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.831268 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.831248 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f793981d-e86b-4866-920e-aa96d4de9e30-signing-key\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.831312 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.831294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95nz\" (UniqueName: \"kubernetes.io/projected/f793981d-e86b-4866-920e-aa96d4de9e30-kube-api-access-k95nz\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.832190 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.832169 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f793981d-e86b-4866-920e-aa96d4de9e30-signing-cabundle\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.839469 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.834029 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f793981d-e86b-4866-920e-aa96d4de9e30-signing-key\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.890199 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.890137 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95nz\" (UniqueName: \"kubernetes.io/projected/f793981d-e86b-4866-920e-aa96d4de9e30-kube-api-access-k95nz\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:17.949917 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:17.949889 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:18.361874 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.361842 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7hrj8"] Oct 30 21:46:18.369011 ip-10-0-137-17 kubenswrapper[2571]: W1030 21:46:18.368985 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf793981d_e86b_4866_920e_aa96d4de9e30.slice/crio-47a8e9fc82ba9e96303e5ffd7abe14ed66c21e46db242d2bf8297aca17b62895 WatchSource:0}: Error finding container 47a8e9fc82ba9e96303e5ffd7abe14ed66c21e46db242d2bf8297aca17b62895: Status 404 returned error can't find the container with id 47a8e9fc82ba9e96303e5ffd7abe14ed66c21e46db242d2bf8297aca17b62895 Oct 30 21:46:18.587102 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.587063 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:18.587479 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.587460 2571 scope.go:122] "RemoveContainer" containerID="c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b" Oct 30 21:46:18.587702 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:18.587677 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jqbm4_openshift-console-operator(04efc6e4-19af-4935-a7c1-2dac67498e4c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" podUID="04efc6e4-19af-4935-a7c1-2dac67498e4c" Oct 30 21:46:18.588074 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.588060 2571 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:18.639941 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.639590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:18.639941 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.639723 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:18.639941 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:18.639805 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:46:18.639941 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:18.639860 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.639842022 +0000 UTC m=+45.603446434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : configmap "kube-rbac-proxy" not found Oct 30 21:46:18.639941 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:18.639890 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:18.640260 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:18.639985 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.639969162 +0000 UTC m=+45.603573575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:18.828355 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.828312 2571 scope.go:122] "RemoveContainer" containerID="c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b" Oct 30 21:46:18.828726 ip-10-0-137-17 kubenswrapper[2571]: E1030 21:46:18.828563 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jqbm4_openshift-console-operator(04efc6e4-19af-4935-a7c1-2dac67498e4c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" podUID="04efc6e4-19af-4935-a7c1-2dac67498e4c" Oct 30 21:46:18.829470 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.829377 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" event={"ID":"f793981d-e86b-4866-920e-aa96d4de9e30","Type":"ContainerStarted","Data":"70ecb4bfc0679349fd625aafe1e7707c4123c3703d516e19a73e5754cf9643df"} Oct 30 21:46:18.829470 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.829424 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" event={"ID":"f793981d-e86b-4866-920e-aa96d4de9e30","Type":"ContainerStarted","Data":"47a8e9fc82ba9e96303e5ffd7abe14ed66c21e46db242d2bf8297aca17b62895"} Oct 30 21:46:18.962576 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:18.962493 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" podStartSLOduration=1.962474947 podStartE2EDuration="1.962474947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:18.959972623 +0000 UTC m=+41.923577074" watchObservedRunningTime="2026-10-30 21:46:18.962474947 +0000 UTC m=+41.926079381" Oct 30 21:46:19.468932 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:19.468914 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g66br_16867cf6-0526-47b2-866e-5dfc13f04869/dns-node-resolver/0.log" Oct 30 21:46:19.641021 ip-10-0-137-17 kubenswrapper[2571]: I1030 21:46:19.640849 2571 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:46:19.641013 ip-10-0-137-17 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:46:19.659671 ip-10-0-137-17 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:46:19.659834 ip-10-0-137-17 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:46:19.659884 ip-10-0-137-17 systemd[1]: kubelet.service: Consumed 3.033s CPU time. Oct 30 21:46:19.670029 ip-10-0-137-17 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:46:19.729847 ip-10-0-137-17 kubenswrapper[4991]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:46:19.729847 ip-10-0-137-17 kubenswrapper[4991]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:46:19.729847 ip-10-0-137-17 kubenswrapper[4991]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:46:19.729847 ip-10-0-137-17 kubenswrapper[4991]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:46:19.731942 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731875 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.731942 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731942 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731947 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731950 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731953 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731957 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731960 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731962 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731971 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731974 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731977 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731979 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731982 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731985 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731990 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731995 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.731998 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732001 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732004 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732007 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.732018 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732009 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732012 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732015 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732017 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732020 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732023 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732026 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732030 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732032 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732035 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732038 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732041 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732044 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732047 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732050 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732052 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732055 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732057 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732060 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.732518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732063 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732065 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732068 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732070 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732073 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732076 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732078 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732081 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732083 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732086 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732089 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732091 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732093 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732096 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732101 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732105 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732109 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732112 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732115 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.732982 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732117 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732120 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732122 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732125 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732127 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732130 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732133 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732136 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732138 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732141 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732144 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732147 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732149 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732152 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732155 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732158 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732160 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732163 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732165 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732168 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.733466 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732171 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732174 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732177 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732180 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732182 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732185 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732188 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732190 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732194 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732196 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732199 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732202 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732204 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732207 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732209 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732212 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732214 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732217 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732220 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732222 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.733963 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732225 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732227 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732230 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732232 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732235 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732237 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732240 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.732242 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733005 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733013 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733017 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733020 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733023 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733025 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733029 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733031 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733034 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733037 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733040 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733042 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.734509 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733045 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733048 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733051 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733054 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733056 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733059 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733062 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733064 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733067 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733069 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733072 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733075 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733077 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733080 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733083 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733085 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733088 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733091 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733093 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.735025 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733097 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733100 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733102 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733105 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733107 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733110 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733113 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733115 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733118 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733122 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733125 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733128 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733132 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733135 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733138 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733141 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733143 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733146 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733149 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.735530 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733151 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733154 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733157 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733159 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733163 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733165 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733168 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733170 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733173 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733176 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733180 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733182 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733185 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733188 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733190 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733193 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733195 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733198 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733201 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733203 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.735992 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733206 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733209 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733211 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733214 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733218 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733222 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733226 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733229 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733232 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733235 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733237 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733240 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733243 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733245 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733248 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733250 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733253 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733255 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733258 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.736518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733260 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733263 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733266 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733269 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733272 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733274 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733277 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733279 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733282 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733284 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733287 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733290 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733293 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733295 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733298 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733300 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733303 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733817 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733826 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733829 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.737016 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733831 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733834 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733837 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733839 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733842 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733851 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733854 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733857 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733859 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733863 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733865 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733868 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733870 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733873 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733882 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733885 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733888 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733891 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733893 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733896 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.737524 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733899 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733902 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733905 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733907 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733911 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733915 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733918 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733920 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733923 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733925 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733930 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733932 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733935 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733938 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733940 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733943 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733946 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733949 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733951 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.738013 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733954 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733957 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733960 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733962 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733965 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733968 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733971 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733973 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733982 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733985 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733987 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733991 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733993 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733996 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.733998 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734001 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734003 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734006 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734009 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734011 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.738847 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734014 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734016 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734019 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734022 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734025 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734027 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734030 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734032 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734035 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734037 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734040 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734044 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734047 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734049 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734052 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734056 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734059 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734062 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734065 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.739787 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734067 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734070 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734083 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734086 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734089 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734091 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734094 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734096 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734099 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734101 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734104 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734106 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734109 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734111 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734114 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734116 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734119 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734122 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734125 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.740565 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734128 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734131 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734134 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734136 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734139 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734143 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734236 4991 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734243 4991 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734255 4991 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734259 4991 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734263 4991 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734267 4991 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734271 4991 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734275 4991 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734279 4991 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734282 4991 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734291 4991 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734295 4991 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734298 4991 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734301 4991 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734304 4991 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734308 4991 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734311 4991 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:46:19.741051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734314 4991 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734338 4991 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734341 4991 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734344 4991 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734348 4991 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734351 4991 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734355 4991 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734358 4991 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734361 4991 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734364 4991 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734367 4991 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734370 4991 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734373 4991 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734376 4991 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734381 4991 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734385 4991 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734388 4991 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734399 4991 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734402 4991 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734405 4991 flags.go:64] FLAG: --enable-server="true" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734408 4991 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734416 4991 flags.go:64] FLAG: --event-burst="100" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734419 4991 flags.go:64] FLAG: --event-qps="50" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734422 4991 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:46:19.741645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734425 4991 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734428 4991 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734433 4991 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734436 4991 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734439 4991 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734442 4991 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734445 4991 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734448 4991 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734451 4991 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734454 4991 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734457 4991 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734460 4991 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734463 4991 flags.go:64] FLAG: --feature-gates="" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734467 4991 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734471 4991 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734474 4991 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734477 4991 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734480 4991 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734483 4991 flags.go:64] FLAG: --help="false" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734486 4991 flags.go:64] FLAG: --hostname-override="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734489 4991 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734492 4991 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734497 4991 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:46:19.742238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734501 4991 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734504 4991 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734507 4991 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734510 4991 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734513 4991 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734516 4991 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734519 4991 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734523 4991 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734526 4991 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734529 4991 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734532 4991 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734535 4991 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734538 4991 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734541 4991 flags.go:64] FLAG: --lock-file="" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734544 4991 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734547 4991 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734550 4991 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734556 4991 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734559 4991 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734562 4991 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734565 4991 flags.go:64] FLAG: --logging-format="text" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734568 4991 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734571 4991 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734574 4991 flags.go:64] FLAG: --manifest-url="" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734577 4991 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:46:19.742816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734582 4991 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734585 4991 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734588 4991 flags.go:64] FLAG: --max-pods="110" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734591 4991 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734594 4991 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734597 4991 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734604 4991 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734607 4991 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734610 4991 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734613 4991 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734620 4991 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734623 4991 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734626 4991 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734629 4991 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734632 4991 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734635 4991 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734638 4991 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734641 4991 flags.go:64] FLAG: --port="10250" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734644 4991 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734647 4991 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01c05a4f2dc47ba5a" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734650 4991 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734653 4991 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734656 4991 flags.go:64] FLAG: --register-node="true" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734659 4991 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734663 4991 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734666 4991 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:46:19.743517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734669 4991 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734672 4991 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734676 4991 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734679 4991 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734682 4991 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734685 4991 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734688 4991 flags.go:64] FLAG: --runonce="false" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734691 4991 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734694 4991 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734697 4991 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734700 4991 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734703 4991 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734708 4991 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734711 4991 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734714 4991 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734717 4991 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734721 4991 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734724 4991 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734727 4991 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734730 4991 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734733 4991 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734736 4991 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734741 4991 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734744 4991 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734747 4991 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:46:19.744157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734754 4991 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734757 4991 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734766 4991 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734769 4991 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734772 4991 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734775 4991 flags.go:64] FLAG: --v="2" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734779 4991 flags.go:64] FLAG: --version="false" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734784 4991 flags.go:64] FLAG: --vmodule="" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734788 4991 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.734791 4991 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734953 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734957 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734960 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734963 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734965 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734968 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734971 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734974 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734976 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734980 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734983 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.744891 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734985 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734988 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734991 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734994 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734996 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.734999 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735002 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735005 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735007 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735010 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735013 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735015 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735018 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735020 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735029 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735032 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735035 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735037 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735042 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.745743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735045 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735048 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735051 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735054 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735056 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735059 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735061 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735064 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735067 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735069 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735072 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735077 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735080 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735083 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735085 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735088 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735091 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735093 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735096 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735099 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.746438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735101 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735104 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735107 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735110 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735112 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735115 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735118 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735120 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735124 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735127 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735129 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735132 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735136 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735138 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735141 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735144 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735146 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735149 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735152 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735155 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.746993 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735157 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735160 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735163 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735167 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735170 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735172 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735175 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735178 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735180 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735183 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735185 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735188 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735191 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735193 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735196 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735199 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735202 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735204 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735207 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735209 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.747620 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735212 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735218 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735221 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735223 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735227 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735230 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735234 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735238 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735241 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735243 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735246 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735249 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735252 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735254 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735257 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.748270 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735261 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.735286 4991 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735346 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735352 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735355 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735358 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735361 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735364 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735367 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735370 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735372 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735377 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735381 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735383 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735386 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735389 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.748807 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735392 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735395 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735397 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735406 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735409 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735413 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735416 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735418 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735421 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735424 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735426 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735429 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735432 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735434 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735437 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735439 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735442 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735449 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735452 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.749290 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735455 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735457 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735460 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735463 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735466 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735468 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735471 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735474 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735476 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735479 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735481 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735484 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735487 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735490 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735492 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735496 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735500 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735508 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735513 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.749785 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735516 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735518 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735521 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735524 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735526 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735529 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735531 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735535 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735537 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735540 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735543 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735547 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735550 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735552 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735555 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735558 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735561 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735564 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735567 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735570 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.750253 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735572 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735575 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735578 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735580 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735583 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735586 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735588 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735591 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735593 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735596 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735599 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735603 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735606 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735608 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735611 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735614 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735616 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735619 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735621 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735624 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.750759 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735626 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735629 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735632 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735636 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735638 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735641 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735644 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735647 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735650 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735652 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735655 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735658 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735660 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.735663 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.740714 4991 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.740728 4991 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.740747 4991 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.740752 4991 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740807 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.751280 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740812 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740816 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740819 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740822 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740824 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740827 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740830 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740833 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740836 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740839 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740842 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740845 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740847 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740850 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740853 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740856 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740859 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740862 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740864 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.751946 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740868 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740870 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740873 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740876 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740879 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740881 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740884 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740887 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740889 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740892 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740894 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740897 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740900 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740902 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740905 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740908 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740910 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740913 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740916 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.752558 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740919 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740922 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740925 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740927 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740930 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740933 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740936 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740938 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740941 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740944 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740948 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740951 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740955 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740958 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740962 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740965 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740968 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740970 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740973 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.753200 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740976 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740978 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740981 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740984 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740987 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740989 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740992 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740995 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.740997 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741000 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741004 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741006 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741009 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741012 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741014 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741017 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741019 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741022 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741025 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741027 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.753997 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741030 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741033 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741035 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741038 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741040 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741044 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741046 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741049 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741051 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741054 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741057 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741060 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741062 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741065 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741068 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741071 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741073 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741076 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741079 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741081 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.754846 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741084 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741086 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741089 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741092 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741094 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741097 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741099 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741103 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.741134 4991 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741277 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741282 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741286 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741289 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741292 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741297 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.755743 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741302 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741305 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741310 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741313 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741331 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741336 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741341 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741344 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741348 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741350 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741353 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741356 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741358 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741361 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741364 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741367 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741370 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741372 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741375 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741377 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.756517 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741380 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741383 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741386 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741388 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741391 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741394 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741397 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741400 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741402 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741405 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741407 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741410 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741414 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741416 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741419 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741422 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741424 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741427 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741429 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.757551 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741432 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741434 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741437 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741440 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741442 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741445 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741447 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741450 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741452 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741455 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741458 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741460 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741463 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741466 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741469 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741471 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741474 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741476 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741479 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741482 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.758521 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741484 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741487 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741489 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741492 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741494 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741497 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741500 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741502 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741505 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741507 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741510 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741513 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741515 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741518 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741520 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741523 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741525 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741528 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741530 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.759556 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741533 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741535 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741538 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741541 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741543 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741548 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741551 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741554 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741556 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741559 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741562 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741565 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741567 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741570 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741572 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741575 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741578 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741580 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741583 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741586 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.760341 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741589 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741592 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.741616 4991 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741647 4991 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741651 4991 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741655 4991 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741658 4991 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741661 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741664 4991 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741667 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741669 4991 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741672 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741675 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741678 4991 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741681 4991 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741683 4991 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:46:19.761276 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741686 4991 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741689 4991 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741693 4991 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741697 4991 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741701 4991 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741704 4991 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741707 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741710 4991 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741713 4991 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741716 4991 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741719 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741722 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741724 4991 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741727 4991 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741730 4991 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741732 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741735 4991 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741738 4991 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741741 4991 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:46:19.762014 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741743 4991 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741746 4991 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741749 4991 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741752 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741754 4991 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741757 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741759 4991 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741762 4991 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741764 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741767 4991 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741770 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741772 4991 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741775 4991 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741778 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741781 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741783 4991 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741786 4991 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741788 4991 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741792 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:46:19.762878 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741794 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741797 4991 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741799 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741802 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741804 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741807 4991 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741809 4991 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741812 4991 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741814 4991 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741817 4991 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741819 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741822 4991 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741826 4991 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741829 4991 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741831 4991 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741834 4991 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741836 4991 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741839 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741842 4991 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741844 4991 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:46:19.763448 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741847 4991 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741849 4991 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741852 4991 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741854 4991 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741857 4991 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741860 4991 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741862 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741865 4991 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741867 4991 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741870 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741873 4991 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741875 4991 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741878 4991 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741880 4991 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741883 4991 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741885 4991 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741888 4991 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741890 4991 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741893 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:46:19.764033 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741896 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741898 4991 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741901 4991 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741903 4991 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741906 4991 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741909 4991 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741912 4991 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741915 4991 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741918 4991 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741920 4991 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741923 4991 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741925 4991 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741928 4991 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741930 4991 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741933 4991 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:19.741936 4991 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.741982 4991 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.742914 4991 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.742966 4991 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:46:19.764600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.743135 4991 server.go:1014] "Starting client certificate rotation" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.743285 4991 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.743357 4991 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:40:37 +0000 UTC" deadline="2028-07-08 10:42:56.723442788 +0000 UTC" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.743409 4991 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14796h56m36.980035922s" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.743619 4991 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.744604 4991 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.746180 4991 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.747881 4991 log.go:25] "Validated CRI v1 image API" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.748626 4991 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:46:19.765088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.750562 4991 fs.go:135] Filesystem UUIDs: map[22729a8c-cada-4251-8364-537976f5e64c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d587f00f-3c55-4d3b-9e2c-d92e16d0704f:/dev/nvme0n1p4] Oct 30 21:46:19.765391 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.750582 4991 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1445753add3bfd08ac88d593db19e31333214b4625d83a4cb70701fc4d2f74b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1445753add3bfd08ac88d593db19e31333214b4625d83a4cb70701fc4d2f74b3/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1626321413d45115210cddc16a6e7171d7331e29e6219f828300aece3186d985/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1626321413d45115210cddc16a6e7171d7331e29e6219f828300aece3186d985/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e8e18cada4240db6dc7842e0cdca1e77cc61bc41b3c961c8425b94e068e030d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e8e18cada4240db6dc7842e0cdca1e77cc61bc41b3c961c8425b94e068e030d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e08256481da327cead75d4879ec52c69fe23a634c3bd64872cc01c3014958c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e08256481da327cead75d4879ec52c69fe23a634c3bd64872cc01c3014958c0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ee752b50093429980c3751f2b20b18a96ee782759cac4150a6654a1f4a5ffb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ee752b50093429980c3751f2b20b18a96ee782759cac4150a6654a1f4a5ffb3/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ef0864688ddab0b08ffbaefe76d5089f9544f86748c885c9e4a73596a7370c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ef0864688ddab0b08ffbaefe76d5089f9544f86748c885c9e4a73596a7370c6/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47a8e9fc82ba9e96303e5ffd7abe14ed66c21e46db242d2bf8297aca17b62895/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47a8e9fc82ba9e96303e5ffd7abe14ed66c21e46db242d2bf8297aca17b62895/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49b2fd0dd2fd5d1086abb56f810fb359086a6ba1020a275a75c10d3381ff6cc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49b2fd0dd2fd5d1086abb56f810fb359086a6ba1020a275a75c10d3381ff6cc2/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bbafb7abf8ec0f03f073c5f55e107eb00f23c3270c1bbdcaed1182a2a28ce99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bbafb7abf8ec0f03f073c5f55e107eb00f23c3270c1bbdcaed1182a2a28ce99/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/509027a15f063ed048e5d0e6cebbc821d5a4e612539b315bc5c57591aeb57ffe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/509027a15f063ed048e5d0e6cebbc821d5a4e612539b315bc5c57591aeb57ffe/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5264bc20d60def2a0748f92e789e9ccaa9102a8306204f3ac4f48631d926c53c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5264bc20d60def2a0748f92e789e9ccaa9102a8306204f3ac4f48631d926c53c/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cf1f65e9e7d6f4ecabbf5decae65ab7f71a07bc289c1619c58a4e58116c233b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cf1f65e9e7d6f4ecabbf5decae65ab7f71a07bc289c1619c58a4e58116c233b/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7525d5af9e663240e9512b9842541a26515277797825ebd2acc26ed5300509b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7525d5af9e663240e9512b9842541a26515277797825ebd2acc26ed5300509b3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7793ccc2ca8bd97467b6023a3f53c8abaa3350e83a2e74fa7ca0c53a95649409/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7793ccc2ca8bd97467b6023a3f53c8abaa3350e83a2e74fa7ca0c53a95649409/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c7bf08e8efb8670722e2a9d6381dd17cbb9d77bc7126502c92f5a45c302ee15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c7bf08e8efb8670722e2a9d6381dd17cbb9d77bc7126502c92f5a45c302ee15/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d13a38f48e0d71f1c0565f93427760ce25fa590e82ed05cfac225dbbc543ae3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d13a38f48e0d71f1c0565f93427760ce25fa590e82ed05cfac225dbbc543ae3/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88f30cdce0e32ff2eb885806ed2cd6a8952f5711092d6a8f84f75fd9c5e7f073/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88f30cdce0e32ff2eb885806ed2cd6a8952f5711092d6a8f84f75fd9c5e7f073/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b8750494f7b96e46fe5038c681ed8e500e6db93da09b8784b689bc296fd5efb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b8750494f7b96e46fe5038c681ed8e500e6db93da09b8784b689bc296fd5efb/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/961818bf1058c55fa1d060bb4f59e064c43d1e75c30844cfa257aeb53528aaf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/961818bf1058c55fa1d060bb4f59e064c43d1e75c30844cfa257aeb53528aaf7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6b8d110b3d8a43cb1de351c753c9e78f1a1071294c70f1f61d2ed8ef4f14400/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6b8d110b3d8a43cb1de351c753c9e78f1a1071294c70f1f61d2ed8ef4f14400/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce78d7cc07629e2740d42c541a2bdf9eb56b89633de63ae123ea6b26d4ea8a3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce78d7cc07629e2740d42c541a2bdf9eb56b89633de63ae123ea6b26d4ea8a3e/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e050a36b4e73089b079cde1c083211d3ce46e7d94b274acf395c651303e3afec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e050a36b4e73089b079cde1c083211d3ce46e7d94b274acf395c651303e3afec/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7bb17a940c0a395c77520a0b2daea86209f8b4890de812cbf53ce7d4581f639/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7bb17a940c0a395c77520a0b2daea86209f8b4890de812cbf53ce7d4581f639/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04efc6e4-19af-4935-a7c1-2dac67498e4c/volumes/kubernetes.io~projected/kube-api-access-q67tn:{mountpoint:/var/lib/kubelet/pods/04efc6e4-19af-4935-a7c1-2dac67498e4c/volumes/kubernetes.io~projected/kube-api-access-q67tn major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04efc6e4-19af-4935-a7c1-2dac67498e4c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/04efc6e4-19af-4935-a7c1-2dac67498e4c/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0abc3e92-4ede-4421-ab8d-91477bd6b85e/volumes/kubernetes.io~projected/kube-api-access-r7xlh:{mountpoint:/var/lib/kubelet/pods/0abc3e92-4ede-4421-ab8d-91477bd6b85e/volumes/kubernetes.io~projected/kube-api-access-r7xlh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~projected/kube-api-access-jhqdn:{mountpoint:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~projected/kube-api-access-jhqdn major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12b9b148-13a4-40f3-9113-2877f5ca9574/volumes/kubernetes.io~projected/kube-api-access-j47qr:{mountpoint:/var/lib/kubelet/pods/12b9b148-13a4-40f3-9113-2877f5ca9574/volumes/kubernetes.io~projected/kube-api-access-j47qr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16867cf6-0526-47b2-866e-5dfc13f04869/volumes/kubernetes.io~projected/kube-api-access-l58vn:{mountpoint:/var/lib/kubelet/pods/16867cf6-0526-47b2-866e-5dfc13f04869/volumes/kubernetes.io~projected/kube-api-access-l58vn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/177b7d0a-d868-497a-b648-23228667c743/volumes/kubernetes.io~projected/kube-api-access-7z7mx:{mountpoint:/var/lib/kubelet/pods/177b7d0a-d868-497a-b648-23228667c743/volumes/kubernetes.io~projected/kube-api-access-7z7mx major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a5249fa-084f-451b-8e94-9cbd497a8ea7/volumes/kubernetes.io~projected/kube-api-access-ntnrc:{mountpoint:/var/lib/kubelet/pods/2a5249fa-084f-451b-8e94-9cbd497a8ea7/volumes/kubernetes.io~projected/kube-api-access-ntnrc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2da4fada-55a5-42ad-b873-46f86929c5d0/volumes/kubernetes.io~projected/kube-api-access-5sz9x:{mountpoint:/var/lib/kubelet/pods/2da4fada-55a5-42ad-b873-46f86929c5d0/volumes/kubernetes.io~projected/kube-api-access-5sz9x major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2da4fada-55a5-42ad-b873-46f86929c5d0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2da4fada-55a5-42ad-b873-46f86929c5d0/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~projected/kube-api-access-sgvtf:{mountpoint:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~projected/kube-api-access-sgvtf major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac87e61-9d62-4657-90a3-2bf7e4877f85/volumes/kubernetes.io~projected/kube-api-access-fxspb:{mountpoint:/var/lib/kubelet/pods/3ac87e61-9d62-4657-90a3-2bf7e4877f85/volumes/kubernetes.io~projected/kube-api-access-fxspb major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f3afb35-bfde-45ec-a972-78a6e7461d9d/volumes/kubernetes.io~projected/kube-api-access-thwfx:{mountpoint:/var/lib/kubelet/pods/4f3afb35-bfde-45ec-a972-78a6e7461d9d/volumes/kubernetes.io~projected/kube-api-access-thwfx major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f3afb35-bfde-45ec-a972-78a6e7461d9d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4f3afb35-bfde-45ec-a972-78a6e7461d9d/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~projected/kube-api-access-f5jtl:{mountpoint:/var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~projected/kube-api-access-f5jtl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/552b3503-9502-4ac7-af98-0acbc947254c/volumes/kubernetes.io~projected/kube-api-access-d82np:{mountpoint:/var/lib/kubelet/pods/552b3503-9502-4ac7-af98-0acbc947254c/volumes/kubernetes.io~projected/kube-api-access-d82np major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/573e461a-8123-4e53-9134-80d6eb9a3e71/volumes/kubernetes.io~projected/kube-api-access-dzk64:{mountpoint:/var/lib/kubelet/pods/573e461a-8123-4e53-9134-80d6eb9a3e71/volumes/kubernetes.io~projected/kube-api-access-dzk64 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5af23954-d911-4664-872d-9ae2e67ecfa0/volumes/kubernetes.io~projected/kube-api-access-cw4qt:{mountpoint:/var/lib/kubelet/pods/5af23954-d911-4664-872d-9ae2e67ecfa0/volumes/kubernetes.io~projected/kube-api-access-cw4qt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a/volumes/kubernetes.io~projected/kube-api-access-cqg42:{mountpoint:/var/lib/kubelet/pods/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a/volumes/kubernetes.io~projected/kube-api-access-cqg42 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4c949e-cff6-4abf-b3ff-f85046f76784/volumes/kubernetes.io~projected/kube-api-access-2pxh4:{mountpoint:/var/lib/kubelet/pods/6b4c949e-cff6-4abf-b3ff-f85046f76784/volumes/kubernetes.io~projected/kube-api-access-2pxh4 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a/volumes/kubernetes.io~projected/kube-api-access-nxtvm:{mountpoint:/var/lib/kubelet/pods/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a/volumes/kubernetes.io~projected/kube-api-access-nxtvm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74d26ea6-81b0-4ff7-acef-cd28751f4a78/volumes/kubernetes.io~projected/kube-api-access-sl44l:{mountpoint:/var/lib/kubelet/pods/74d26ea6-81b0-4ff7-acef-cd28751f4a78/volumes/kubernetes.io~projected/kube-api-access-sl44l major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e021343-49b5-4868-a1f8-3a1bd283c42c/volumes/kubernetes.io~projected/kube-api-access-2fnkh:{mountpoint:/var/lib/kubelet/pods/7e021343-49b5-4868-a1f8-3a1bd283c42c/volumes/kubernetes.io~projected/kube-api-access-2fnkh major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e37f363-3bfe-4f18-a5ee-f60f849499d1/volumes/kubernetes.io~projected/kube-api-access-7bd5v:{mountpoint:/var/lib/kubelet/pods/7e37f363-3bfe-4f18-a5ee-f60f849499d1/volumes/kubernetes.io~projected/kube-api-access-7bd5v major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8743ad27-5d90-4b7f-8012-6faf21548f97/volumes/kubernetes.io~projected/kube-api-access-7s2xf:{mountpoint:/var/lib/kubelet/pods/8743ad27-5d90-4b7f-8012-6faf21548f97/volumes/kubernetes.io~projected/kube-api-access-7s2xf major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07200b0-a7f6-4c5a-9197-b73779a06c1f/volumes/kubernetes.io~projected/kube-api-access-rpcqn:{mountpoint:/var/lib/kubelet/pods/a07200b0-a7f6-4c5a-9197-b73779a06c1f/volumes/kubernetes.io~projected/kube-api-access-rpcqn major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07200b0-a7f6-4c5a-9197-b73779a06c1f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a07200b0-a7f6-4c5a-9197-b73779a06c1f/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~projected/kube-api-access-7nq4g:{mountpoint:/var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~projected/kube-api-access-7nq4g major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~secret/default-certificate major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6eb7759-f63d-4817-9128-adcecdceb0d7/volumes/kubernetes.io~projected/kube-api-access-45q9q:{mountpoint:/var/lib/kubelet/pods/c6eb7759-f63d-4817-9128-adcecdceb0d7/volumes/kubernetes.io~projected/kube-api-access-45q9q major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3280f18-2d95-4303-8f34-5a2fe91671e6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d3280f18-2d95-4303-8f34-5a2fe91671e6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6594a6f-9c78-40bf-99b7-fb21cd031157/volumes/kubernetes.io~projected/kube-api-access-rtqd5:{mountpoint:/var/lib/kubelet/pods/d6594a6f-9c78-40bf-99b7-fb21cd031157/volumes/kubernetes.io~projected/kube-api-access-rtqd5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6594a6f-9c78-40bf-99b7-fb21cd031157/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d6594a6f-9c78-40bf-99b7-fb21cd031157/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b2124f-2877-4c7b-8fc1-75d3bda8c459/volumes/kubernetes.io~projected/kube-api-access-vjtl9:{mountpoint:/var/lib/kubelet/pods/d9b2124f-2877-4c7b-8fc1-75d3bda8c459/volumes/kubernetes.io~projected/kube-api-access-vjtl9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbdaddf0-875f-46f2-8418-62377b63a086/volumes/kubernetes.io~projected/kube-api-access-22qrq:{mountpoint:/var/lib/kubelet/pods/dbdaddf0-875f-46f2-8418-62377b63a086/volumes/kubernetes.io~projected/kube-api-access-22qrq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volumes/kubernetes.io~projected/kube-api-access-lkccl:{mountpoint:/var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volumes/kubernetes.io~projected/kube-api-access-lkccl major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f793981d-e86b-4866-920e-aa96d4de9e30/volumes/kubernetes.io~projected/kube-api-access-k95nz:{mountpoint:/var/lib/kubelet/pods/f793981d-e86b-4866-920e-aa96d4de9e30/volumes/kubernetes.io~projected/kube-api-access-k95nz major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f793981d-e86b-4866-920e-aa96d4de9e30/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f793981d-e86b-4866-920e-aa96d4de9e30/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbd83615-d417-4d88-88d2-9f74ffbce102/volumes/kubernetes.io~projected/kube-api-access-jcdh8:{mountpoint:/var/lib/kubelet/pods/fbd83615-d417-4d88-88d2-9f74ffbce102/volumes/kubernetes.io~projected/kube-api-access-jcdh8 major:0 minor:309 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/43274fe66a5acc987e2033cece4ffe0acedd7cb36b08ebadd2edf0565d15a1c8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/70792c1884a25a976d8ad3786f9466d2864131513303335d74d0f8ba11d20248/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4a3675874cf23d3a44bb69526bc730344e16ac8000b243aa9178ff28768031c0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/746249c802065bc9d391eaa66782cc9c5733e63c0955cecc67742459d256ad97/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1927a41a54b37c4a920a826b784e3119af28793485559d6f7e9c28b3f390d573/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fb05627c5508c42a9469f9ecbcb8ecd0d2250e8e4e350f43f6e9cf7c475d2755/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2ac8d3f9299af9ec6322a83b5ddc8d6383754ec553bbfbfb92bf6fe1b6c71262/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6923dbb40e5ce67619f5c18a4cd882e8ae51c83876b0838345d1867842163acb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/babe49b794e878609864ccf34cba1dedc5c51faeeaeade55dd3cdcfbdbb4b877/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9b7adafbbf7e889f38065b585846f67518fbd5ff75fa51fd3ca24f0848afddc7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a393d9587dc91aba23e782721613bf1d013c477a52f15610625511a9e04ca09b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c57a1571d7b28f829993072cde62fe545c0aeb9490e3eba1bd2ca30863394ac5/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7a98687089b8d168f44eb59ad24d1e72d68953acd14105ed0d354b65cb093956/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5e4887bcaf7a1257f7cc96deb077bdb3e86643e5203023c6304f575025974aeb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a9447c0f182faf5216eade9eaf7094ab5caaafa487939e4ffe7fbde146b2c16e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f5b61a64d5aedba2e4b3d45626f809f6260c908e3cb12e3351605424d21abd4f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bdc182969b1f9b9df82d4c9e016e9f3f2f921ecdcb180a911ab3f1d58f6f03c5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/00e4b36981c362c2a80e33551ee9a3c6225404c573914f789f7cfebd1cef7b78/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8cfbd75a2a628533e4f0d08f885fd805d805935453b65314fddc4f8f38cee9e3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2ed25c8d2ca1834c7660db28b1f8050757c22ba6af9befedd818cb7939a131f5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ca5175b35c0e3ff006dd58ab8748bb6829b76793c6a1aca431e9cf45fa7bc259/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/33ab3c3b1d7d6672b4e428b931875476db4efafa84be972894428020c8fbd675/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5657b00bfa3bf338bb0d302c8e967f6d3cf1eb572c5c3eb48a797c7439377a60/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/023661a47e8d7be5363d52dd77142086c1fd3b71c0073099bf56bfa58ebb349f/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/89b73f9389e169f5921cfbb94f744f25cdb5997b9b4596bcaf03c0d137f8a5f0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d2dc0dd003d94e46a52bcb96aaf6d422ff44aece6305b8a49279b6ea03716fa3/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/b8f26203c36ced2a840a8fd78a18f054ca38bc77765a29b7e7e3bf6c6eec226f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a423c245bbe89fbe376fe225fc8477766522e6f6e702b6474211aef5b589b611/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/0160d1aa6b50ef4e2ebb65a2aa320793e3f603d5825ccca5ba83750a1adf4144/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/8196d00ecefc3d2f217ce0b0d99b52b938d2cdf93255e3993459eeb236f8e693/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/3e55597394b05d34726386ebf6a7a01ab4dc9f239ce2df29daa095178ac96565/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/067095c72dcf604588171cf88c943d1593551b0380dcc8dea138c269a29dc652/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/1a2b6b8c0ad7e556d46df55b40648515f4ba06c34c8e2ffee47c6fee6104d568/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/10e5fc9c45fd527d8fd8802cfb23ee734ccae35ad0858c5e41b38527ddfac793/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/de8c4602f626ba46e6bd6c8b64130b7182457ec77f4dceb6cf4f3496be1091d2/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/0a058ab07e7dc7d63d81fa409cfaa3b0c797a2e85b5891521d4385f2e2e2d736/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/c6289847879e6b62b91a088854f81ebd16060b82306e22178bebb91909c5c633/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/99e6698d0a9d8640215a1135d5bee75faca97cb17a01ac4db2f256803ef33c16/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/b09543650a29500cbe97f8ca71fd0fc88151a473e43a4180d38022491d556210/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c2e263de52ce556859aa1d3789439cc860884574da2791347f4ff34b9bc5335e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/c855ebc1217febf6a903523add3bf282b768f84329817826e0f0432f92d85eba/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/d14aabb4357a599e4d9ce4f98f1fd4fef05bf338ea0533b20d29b917b54f5d18/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/a916397b6d57ec4ecbc35a585833bb5d3247ef407a256bc6eee993ee138c17fb/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/c39960d5d87a107bab7bd2d1fa2f48367a449faae909a1d5aa6055864c1cb4ea/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/395faae733b9a8df0caad219b369ad08ebcbb6d91bd41b9f51241a53683a2db1/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/2f01c17c2138aa63a4f7c5e8cc317c40568bf5b6e5db07acc8d12d75c6564234/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/13c73023161077b8d2df8626ec9afc9b595ce945eea6dd23eaae9040c5298057/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/c7c5871def0b1ca86c29ac9928e36a5656eef0d856ce746fd8b448fae38edec8/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/5ba92451a488bf9cb2d3a19d69bb1debb30ecf514b231ab8de76288f9cdcda72/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/4a77e7ba3adfb7b7b10af69ab3d4003a4b9896c749dc2b0e6438330157d1daaa/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/53b071c9049a287c996dd5c4a9ce17e15f563b38a413a6ec5e3f586748374869/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/00b508d300ee6ffb7c653e92f2b4fd12f866a9715252b48c47cd7786c29b9a2c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5af74232a20c59319940b2eb95b2baaa3a8ce19b6d30921f521ad39e9b44de23/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dbc58115ce097359a945c90b5973cd51082623758119b2a6d6e372df9a599723/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/679cbe10a8842b9ea7593114555f7288d56b93973318a77fc88a25040fc2f943/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/875b308e0834b9fc7eb9d29d75d4be65245d9cf976411d49ddb989f9e56b52f3/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5fcc9911b71fc04382331386f0076231371262edd2e605f37a4f94418b21c536/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/a95379186758b452a94f93396774d5536e61c13d84417a9934e8a3c966a3dd07/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1a8975a9baa4fb79a4f00c2320df7edbbc1f72ed3704a85afe97430a422d6bc1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/edefbba23a576da9f59374aea064624e3c5aaa5e10c53047c58211e4482b05c6/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.762232 4991 manager.go:217] Machine: {Timestamp:2026-10-30 21:46:19.761443152 +0000 UTC m=+0.060853025 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3141587 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2958a22a3aae2ccad3d305d5a952d0 SystemUUID:ec2958a2-2a3a-ae2c-cad3-d305d5a952d0 BootID:7977c813-1676-484b-8ad6-63067469be66 Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6eb7759-f63d-4817-9128-adcecdceb0d7/volumes/kubernetes.io~projected/kube-api-access-45q9q DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/04efc6e4-19af-4935-a7c1-2dac67498e4c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce78d7cc07629e2740d42c541a2bdf9eb56b89633de63ae123ea6b26d4ea8a3e/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1626321413d45115210cddc16a6e7171d7331e29e6219f828300aece3186d985/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49b2fd0dd2fd5d1086abb56f810fb359086a6ba1020a275a75c10d3381ff6cc2/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1445753add3bfd08ac88d593db19e31333214b4625d83a4cb70701fc4d2f74b3/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3280f18-2d95-4303-8f34-5a2fe91671e6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c7bf08e8efb8670722e2a9d6381dd17cbb9d77bc7126502c92f5a45c302ee15/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e050a36b4e73089b079cde1c083211d3ce46e7d94b274acf395c651303e3afec/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~projected/kube-api-access-f5jtl DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/961818bf1058c55fa1d060bb4f59e064c43d1e75c30844cfa257aeb53528aaf7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~projected/kube-api-access-sgvtf DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12b9b148-13a4-40f3-9113-2877f5ca9574/volumes/kubernetes.io~projected/kube-api-access-j47qr DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2da4fada-55a5-42ad-b873-46f86929c5d0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f3afb35-bfde-45ec-a972-78a6e7461d9d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~projected/kube-api-access-7nq4g DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/509027a15f063ed048e5d0e6cebbc821d5a4e612539b315bc5c57591aeb57ffe/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e8e18cada4240db6dc7842e0cdca1e77cc61bc41b3c961c8425b94e068e030d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6594a6f-9c78-40bf-99b7-fb21cd031157/volumes/kubernetes.io~projected/kube-api-access-rtqd5 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6b8d110b3d8a43cb1de351c753c9e78f1a1071294c70f1f61d2ed8ef4f14400/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d13a38f48e0d71f1c0565f93427760ce25fa590e82ed05cfac225dbbc543ae3/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b8750494f7b96e46fe5038c681ed8e500e6db93da09b8784b689bc296fd5efb/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7525d5af9e663240e9512b9842541a26515277797825ebd2acc26ed5300509b3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a07200b0-a7f6-4c5a-9197-b73779a06c1f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbd83615-d417-4d88-88d2-9f74ffbce102/volumes/kubernetes.io~projected/kube-api-access-jcdh8 DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cf1f65e9e7d6f4ecabbf5decae65ab7f71a07bc289c1619c58a4e58116c233b/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a07200b0-a7f6-4c5a-9197-b73779a06c1f/volumes/kubernetes.io~projected/kube-api-access-rpcqn DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f793981d-e86b-4866-920e-aa96d4de9e30/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a/volumes/kubernetes.io~projected/kube-api-access-nxtvm DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a/volumes/kubernetes.io~projected/kube-api-access-cqg42 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5264bc20d60def2a0748f92e789e9ccaa9102a8306204f3ac4f48631d926c53c/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88f30cdce0e32ff2eb885806ed2cd6a8952f5711092d6a8f84f75fd9c5e7f073/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7bb17a940c0a395c77520a0b2daea86209f8b4890de812cbf53ce7d4581f639/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4c949e-cff6-4abf-b3ff-f85046f76784/volumes/kubernetes.io~projected/kube-api-access-2pxh4 DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ee752b50093429980c3751f2b20b18a96ee782759cac4150a6654a1f4a5ffb3/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/177b7d0a-d868-497a-b648-23228667c743/volumes/kubernetes.io~projected/kube-api-access-7z7mx DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac87e61-9d62-4657-90a3-2bf7e4877f85/volumes/kubernetes.io~projected/kube-api-access-fxspb DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74d26ea6-81b0-4ff7-acef-cd28751f4a78/volumes/kubernetes.io~projected/kube-api-access-sl44l DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2a5249fa-084f-451b-8e94-9cbd497a8ea7/volumes/kubernetes.io~projected/kube-api-access-ntnrc DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e021343-49b5-4868-a1f8-3a1bd283c42c/volumes/kubernetes.io~projected/kube-api-access-2fnkh DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50406e5f-7f63-49e5-aca1-d572b155292a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f008c236-2573-46f0-bbdd-f10daa5d08bf/volumes/kubernetes.io~projected/kube-api-access-lkccl DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f3afb35-bfde-45ec-a972-78a6e7461d9d/volumes/kubernetes.io~projected/kube-api-access-thwfx DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbdaddf0-875f-46f2-8418-62377b63a086/volumes/kubernetes.io~projected/kube-api-access-22qrq DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e08256481da327cead75d4879ec52c69fe23a634c3bd64872cc01c3014958c0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e37f363-3bfe-4f18-a5ee-f60f849499d1/volumes/kubernetes.io~projected/kube-api-access-7bd5v DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9b2124f-2877-4c7b-8fc1-75d3bda8c459/volumes/kubernetes.io~projected/kube-api-access-vjtl9 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0abc3e92-4ede-4421-ab8d-91477bd6b85e/volumes/kubernetes.io~projected/kube-api-access-r7xlh DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes/kubernetes.io~projected/kube-api-access-jhqdn DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ef0864688ddab0b08ffbaefe76d5089f9544f86748c885c9e4a73596a7370c6/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7793ccc2ca8bd97467b6023a3f53c8abaa3350e83a2e74fa7ca0c53a95649409/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bbafb7abf8ec0f03f073c5f55e107eb00f23c3270c1bbdcaed1182a2a28ce99/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f793981d-e86b-4866-920e-aa96d4de9e30/volumes/kubernetes.io~projected/kube-api-access-k95nz DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16867cf6-0526-47b2-866e-5dfc13f04869/volumes/kubernetes.io~projected/kube-api-access-l58vn DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6594a6f-9c78-40bf-99b7-fb21cd031157/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8743ad27-5d90-4b7f-8012-6faf21548f97/volumes/kubernetes.io~projected/kube-api-access-7s2xf DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2da4fada-55a5-42ad-b873-46f86929c5d0/volumes/kubernetes.io~projected/kube-api-access-5sz9x DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/04efc6e4-19af-4935-a7c1-2dac67498e4c/volumes/kubernetes.io~projected/kube-api-access-q67tn DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/573e461a-8123-4e53-9134-80d6eb9a3e71/volumes/kubernetes.io~projected/kube-api-access-dzk64 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a547a8cd-8909-475e-a03d-15b409877777/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47a8e9fc82ba9e96303e5ffd7abe14ed66c21e46db242d2bf8297aca17b62895/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/552b3503-9502-4ac7-af98-0acbc947254c/volumes/kubernetes.io~projected/kube-api-access-d82np DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5af23954-d911-4664-872d-9ae2e67ecfa0/volumes/kubernetes.io~projected/kube-api-access-cw4qt DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1445753add3bfd0 MacAddress:b2:dd:00:bf:88:df Speed:10000 Mtu:8901} {Name:3ee752b50093429 MacAddress:5e:e3:b5:39:41:51 Speed:10000 Mtu:8901} {Name:3ef0864688ddab0 MacAddress:f2:1e:b2:ec:01:f3 Speed:10000 Mtu:8901} {Name:47a8e9fc82ba9e9 MacAddress:c2:0e:2b:b7:ce:30 Speed:10000 Mtu:8901} {Name:49b2fd0dd2fd5d1 MacAddress:a2:d1:49:70:52:8a Speed:10000 Mtu:8901} {Name:4bbafb7abf8ec0f MacAddress:46:3e:0f:94:a9:6f Speed:10000 Mtu:8901} {Name:5264bc20d60def2 MacAddress:fa:e6:4d:b9:f0:39 Speed:10000 Mtu:8901} {Name:5b2834773e61d38 MacAddress:06:71:5e:d5:b3:4a Speed:10000 Mtu:8901} {Name:6cf1f65e9e7d6f4 MacAddress:3a:ae:fc:dd:b2:fc Speed:10000 Mtu:8901} {Name:7c7bf08e8efb867 MacAddress:2a:a5:7e:7b:1a:05 Speed:10000 Mtu:8901} {Name:7d13a38f48e0d71 MacAddress:36:0d:37:91:97:0b Speed:10000 Mtu:8901} {Name:8b8750494f7b96e MacAddress:ae:35:6f:61:ef:63 Speed:10000 Mtu:8901} {Name:b6b8d110b3d8a43 MacAddress:7e:fc:cd:69:6f:88 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f7:05:6d:39 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:0e:a7:cc:50:b2 Speed:0 Mtu:8901} {Name:ce78d7cc07629e2 MacAddress:ee:a7:b8:56:4a:27 Speed:10000 Mtu:8901} {Name:e050a36b4e73089 MacAddress:9e:7b:b9:cc:af:30 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f7:05:6d:39 Speed:0 Mtu:9001} {Name:f7bb17a940c0a39 MacAddress:be:60:0b:56:45:71 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:86:b0:1e:cb:26:08 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:c8:40:19:ae:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.762886 4991 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.762942 4991 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763195 4991 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763222 4991 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-17.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763354 4991 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763363 4991 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763373 4991 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763385 4991 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763403 4991 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763545 4991 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763640 4991 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763677 4991 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763689 4991 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763699 4991 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763708 4991 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.763716 4991 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.764389 4991 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.764399 4991 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:46:19.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.764822 4991 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:46:19.826531 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826515 4991 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:46:19.826531 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826529 4991 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:46:19.826641 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826621 4991 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:46:19.826826 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826815 4991 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:46:19.826857 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826844 4991 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:46:19.826942 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826934 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:46:19.826972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826948 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:46:19.826972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826954 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:46:19.826972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826961 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:46:19.826972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826966 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:46:19.826972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826972 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826978 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826983 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826990 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.826996 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.827010 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.827021 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.827050 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:46:19.827113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.827056 4991 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:46:19.830467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.830452 4991 server.go:1263] "Started kubelet" Oct 30 21:46:19.832613 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.830673 4991 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:46:19.832613 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.831055 4991 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:46:19.832613 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.831149 4991 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:46:19.832613 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.831437 4991 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:46:19.831094 ip-10-0-137-17 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:46:19.836356 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.833891 4991 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:46:19.838851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.838525 4991 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:46:19.838851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.838565 4991 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:46:19.838851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.838605 4991 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:39 +0000 UTC" deadline="2028-07-11 11:30:39.533447934 +0000 UTC" Oct 30 21:46:19.838851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.838622 4991 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14869h44m19.694829217s" Oct 30 21:46:19.838851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.838678 4991 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:46:19.838851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.838701 4991 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:46:19.838851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.838756 4991 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:46:19.839238 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:19.838910 4991 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:46:19.839801 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.839781 4991 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:46:19.839801 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.839801 4991 factory.go:55] Registering systemd factory Oct 30 21:46:19.839969 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.839813 4991 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:46:19.840672 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.840648 4991 factory.go:153] Registering CRI-O factory Oct 30 21:46:19.840755 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.840676 4991 factory.go:223] Registration of the crio container factory successfully Oct 30 21:46:19.840755 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.840698 4991 factory.go:103] Registering Raw factory Oct 30 21:46:19.840755 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.840721 4991 manager.go:1199] Started watching for new ooms in manager Oct 30 21:46:19.841151 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841136 4991 manager.go:319] Starting recovery of all containers Oct 30 21:46:19.841558 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841515 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04efc6e4-19af-4935-a7c1-2dac67498e4c" volumeName="kubernetes.io/secret/04efc6e4-19af-4935-a7c1-2dac67498e4c-serving-cert" seLinuxMountContext="" Oct 30 21:46:19.841640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841563 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" volumeName="kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:46:19.841640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841579 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39391909-c450-41dd-bf4a-51208f0e7c32" volumeName="kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates" seLinuxMountContext="" Oct 30 21:46:19.841640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841597 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39391909-c450-41dd-bf4a-51208f0e7c32" volumeName="kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:46:19.841640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841614 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67d09fe6-fb0a-43cc-8b20-98cdb22eea2a" volumeName="kubernetes.io/projected/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-kube-api-access-cqg42" seLinuxMountContext="" Oct 30 21:46:19.841640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841627 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07200b0-a7f6-4c5a-9197-b73779a06c1f" volumeName="kubernetes.io/projected/a07200b0-a7f6-4c5a-9197-b73779a06c1f-kube-api-access-rpcqn" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841643 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6594a6f-9c78-40bf-99b7-fb21cd031157" volumeName="kubernetes.io/secret/d6594a6f-9c78-40bf-99b7-fb21cd031157-serving-cert" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841656 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" volumeName="kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841676 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="573e461a-8123-4e53-9134-80d6eb9a3e71" volumeName="kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841693 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4c949e-cff6-4abf-b3ff-f85046f76784" volumeName="kubernetes.io/projected/6b4c949e-cff6-4abf-b3ff-f85046f76784-kube-api-access-2pxh4" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841706 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8743ad27-5d90-4b7f-8012-6faf21548f97" volumeName="kubernetes.io/empty-dir/8743ad27-5d90-4b7f-8012-6faf21548f97-tmp-dir" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841745 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8743ad27-5d90-4b7f-8012-6faf21548f97" volumeName="kubernetes.io/projected/8743ad27-5d90-4b7f-8012-6faf21548f97-kube-api-access-7s2xf" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841760 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6eb7759-f63d-4817-9128-adcecdceb0d7" volumeName="kubernetes.io/projected/c6eb7759-f63d-4817-9128-adcecdceb0d7-kube-api-access-45q9q" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841777 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbdaddf0-875f-46f2-8418-62377b63a086" volumeName="kubernetes.io/projected/dbdaddf0-875f-46f2-8418-62377b63a086-kube-api-access-22qrq" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841798 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04efc6e4-19af-4935-a7c1-2dac67498e4c" volumeName="kubernetes.io/projected/04efc6e4-19af-4935-a7c1-2dac67498e4c-kube-api-access-q67tn" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841817 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" volumeName="kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841833 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" volumeName="kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841845 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16867cf6-0526-47b2-866e-5dfc13f04869" volumeName="kubernetes.io/empty-dir/16867cf6-0526-47b2-866e-5dfc13f04869-tmp-dir" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841856 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07200b0-a7f6-4c5a-9197-b73779a06c1f" volumeName="kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-snapshots" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841868 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a547a8cd-8909-475e-a03d-15b409877777" volumeName="kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-stats-auth" seLinuxMountContext="" Oct 30 21:46:19.841878 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841880 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad6427c-0014-4c83-9ff9-ce1d9ba0aa55" volumeName="kubernetes.io/configmap/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-konnectivity-ca" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841892 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39391909-c450-41dd-bf4a-51208f0e7c32" volumeName="kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841904 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74d26ea6-81b0-4ff7-acef-cd28751f4a78" volumeName="kubernetes.io/projected/74d26ea6-81b0-4ff7-acef-cd28751f4a78-kube-api-access-sl44l" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841915 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a547a8cd-8909-475e-a03d-15b409877777" volumeName="kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-default-certificate" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841926 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b2124f-2877-4c7b-8fc1-75d3bda8c459" volumeName="kubernetes.io/projected/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kube-api-access-vjtl9" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841938 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5249fa-084f-451b-8e94-9cbd497a8ea7" volumeName="kubernetes.io/projected/2a5249fa-084f-451b-8e94-9cbd497a8ea7-kube-api-access-ntnrc" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841964 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5af23954-d911-4664-872d-9ae2e67ecfa0" volumeName="kubernetes.io/projected/5af23954-d911-4664-872d-9ae2e67ecfa0-kube-api-access-cw4qt" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.841992 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07200b0-a7f6-4c5a-9197-b73779a06c1f" volumeName="kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-service-ca-bundle" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842008 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6594a6f-9c78-40bf-99b7-fb21cd031157" volumeName="kubernetes.io/projected/d6594a6f-9c78-40bf-99b7-fb21cd031157-kube-api-access-rtqd5" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842020 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" volumeName="kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842032 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da4fada-55a5-42ad-b873-46f86929c5d0" volumeName="kubernetes.io/projected/2da4fada-55a5-42ad-b873-46f86929c5d0-kube-api-access-5sz9x" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842044 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" volumeName="kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842056 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="177b7d0a-d868-497a-b648-23228667c743" volumeName="kubernetes.io/projected/177b7d0a-d868-497a-b648-23228667c743-kube-api-access-7z7mx" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842068 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50406e5f-7f63-49e5-aca1-d572b155292a" volumeName="kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-etc-tuned" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842080 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8743ad27-5d90-4b7f-8012-6faf21548f97" volumeName="kubernetes.io/configmap/8743ad27-5d90-4b7f-8012-6faf21548f97-config-volume" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842091 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3280f18-2d95-4303-8f34-5a2fe91671e6" volumeName="kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842103 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f008c236-2573-46f0-bbdd-f10daa5d08bf" volumeName="kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-env-overrides" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842114 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16867cf6-0526-47b2-866e-5dfc13f04869" volumeName="kubernetes.io/projected/16867cf6-0526-47b2-866e-5dfc13f04869-kube-api-access-l58vn" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842126 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da4fada-55a5-42ad-b873-46f86929c5d0" volumeName="kubernetes.io/secret/2da4fada-55a5-42ad-b873-46f86929c5d0-cert" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842139 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39391909-c450-41dd-bf4a-51208f0e7c32" volumeName="kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842150 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c494bc9-5908-48bc-9825-2669ad4196c1" volumeName="kubernetes.io/configmap/4c494bc9-5908-48bc-9825-2669ad4196c1-nginx-conf" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842161 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50406e5f-7f63-49e5-aca1-d572b155292a" volumeName="kubernetes.io/projected/50406e5f-7f63-49e5-aca1-d572b155292a-kube-api-access-f5jtl" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842174 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5af23954-d911-4664-872d-9ae2e67ecfa0" volumeName="kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-cni-binary-copy" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842186 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39391909-c450-41dd-bf4a-51208f0e7c32" volumeName="kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842197 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39391909-c450-41dd-bf4a-51208f0e7c32" volumeName="kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842210 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67d09fe6-fb0a-43cc-8b20-98cdb22eea2a" volumeName="kubernetes.io/configmap/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-telemetry-config" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842223 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07200b0-a7f6-4c5a-9197-b73779a06c1f" volumeName="kubernetes.io/secret/a07200b0-a7f6-4c5a-9197-b73779a06c1f-serving-cert" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842248 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f008c236-2573-46f0-bbdd-f10daa5d08bf" volumeName="kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842263 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5249fa-084f-451b-8e94-9cbd497a8ea7" volumeName="kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842275 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac87e61-9d62-4657-90a3-2bf7e4877f85" volumeName="kubernetes.io/projected/3ac87e61-9d62-4657-90a3-2bf7e4877f85-kube-api-access-fxspb" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842297 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f3afb35-bfde-45ec-a972-78a6e7461d9d" volumeName="kubernetes.io/configmap/4f3afb35-bfde-45ec-a972-78a6e7461d9d-config" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842311 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07200b0-a7f6-4c5a-9197-b73779a06c1f" volumeName="kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842351 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a547a8cd-8909-475e-a03d-15b409877777" volumeName="kubernetes.io/projected/a547a8cd-8909-475e-a03d-15b409877777-kube-api-access-7nq4g" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842363 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ffd4edc-ce40-4a48-b398-c2d907a7cd0a" volumeName="kubernetes.io/projected/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a-kube-api-access-nxtvm" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842375 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e021343-49b5-4868-a1f8-3a1bd283c42c" volumeName="kubernetes.io/projected/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-api-access-2fnkh" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842387 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39391909-c450-41dd-bf4a-51208f0e7c32" volumeName="kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842416 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5249fa-084f-451b-8e94-9cbd497a8ea7" volumeName="kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-binary-copy" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842437 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f3afb35-bfde-45ec-a972-78a6e7461d9d" volumeName="kubernetes.io/projected/4f3afb35-bfde-45ec-a972-78a6e7461d9d-kube-api-access-thwfx" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842450 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="552b3503-9502-4ac7-af98-0acbc947254c" volumeName="kubernetes.io/configmap/552b3503-9502-4ac7-af98-0acbc947254c-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842463 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f793981d-e86b-4866-920e-aa96d4de9e30" volumeName="kubernetes.io/projected/f793981d-e86b-4866-920e-aa96d4de9e30-kube-api-access-k95nz" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842476 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad6427c-0014-4c83-9ff9-ce1d9ba0aa55" volumeName="kubernetes.io/secret/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-agent-certs" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842487 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" volumeName="kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842498 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5af23954-d911-4664-872d-9ae2e67ecfa0" volumeName="kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-daemon-config" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842510 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e021343-49b5-4868-a1f8-3a1bd283c42c" volumeName="kubernetes.io/empty-dir/7e021343-49b5-4868-a1f8-3a1bd283c42c-data-volume" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842522 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f008c236-2573-46f0-bbdd-f10daa5d08bf" volumeName="kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-config" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842535 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbd83615-d417-4d88-88d2-9f74ffbce102" volumeName="kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842546 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbd83615-d417-4d88-88d2-9f74ffbce102" volumeName="kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842559 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0abc3e92-4ede-4421-ab8d-91477bd6b85e" volumeName="kubernetes.io/projected/0abc3e92-4ede-4421-ab8d-91477bd6b85e-kube-api-access-r7xlh" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842571 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b9b148-13a4-40f3-9113-2877f5ca9574" volumeName="kubernetes.io/configmap/12b9b148-13a4-40f3-9113-2877f5ca9574-serviceca" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842583 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e37f363-3bfe-4f18-a5ee-f60f849499d1" volumeName="kubernetes.io/projected/7e37f363-3bfe-4f18-a5ee-f60f849499d1-kube-api-access-7bd5v" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842595 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f793981d-e86b-4866-920e-aa96d4de9e30" volumeName="kubernetes.io/configmap/f793981d-e86b-4866-920e-aa96d4de9e30-signing-cabundle" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842606 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f793981d-e86b-4866-920e-aa96d4de9e30" volumeName="kubernetes.io/secret/f793981d-e86b-4866-920e-aa96d4de9e30-signing-key" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842617 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04efc6e4-19af-4935-a7c1-2dac67498e4c" volumeName="kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-trusted-ca" seLinuxMountContext="" Oct 30 21:46:19.842600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842628 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b9b148-13a4-40f3-9113-2877f5ca9574" volumeName="kubernetes.io/projected/12b9b148-13a4-40f3-9113-2877f5ca9574-kube-api-access-j47qr" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842641 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f3afb35-bfde-45ec-a972-78a6e7461d9d" volumeName="kubernetes.io/secret/4f3afb35-bfde-45ec-a972-78a6e7461d9d-serving-cert" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842654 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="552b3503-9502-4ac7-af98-0acbc947254c" volumeName="kubernetes.io/projected/552b3503-9502-4ac7-af98-0acbc947254c-kube-api-access-d82np" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842666 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbd83615-d417-4d88-88d2-9f74ffbce102" volumeName="kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842677 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5249fa-084f-451b-8e94-9cbd497a8ea7" volumeName="kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842688 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50406e5f-7f63-49e5-aca1-d572b155292a" volumeName="kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-tmp" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842700 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4c949e-cff6-4abf-b3ff-f85046f76784" volumeName="kubernetes.io/configmap/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-config" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842711 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07200b0-a7f6-4c5a-9197-b73779a06c1f" volumeName="kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-tmp" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842722 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6594a6f-9c78-40bf-99b7-fb21cd031157" volumeName="kubernetes.io/configmap/d6594a6f-9c78-40bf-99b7-fb21cd031157-config" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842734 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f008c236-2573-46f0-bbdd-f10daa5d08bf" volumeName="kubernetes.io/secret/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842745 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbd83615-d417-4d88-88d2-9f74ffbce102" volumeName="kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842756 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04efc6e4-19af-4935-a7c1-2dac67498e4c" volumeName="kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-config" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842767 4991 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f008c236-2573-46f0-bbdd-f10daa5d08bf" volumeName="kubernetes.io/projected/f008c236-2573-46f0-bbdd-f10daa5d08bf-kube-api-access-lkccl" seLinuxMountContext="" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842781 4991 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:46:19.844988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.842789 4991 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:46:19.845614 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.845601 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:46:19.847713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.847694 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:46:19.852618 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.852587 4991 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:46:19.866816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.866799 4991 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:46:19.866816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.866818 4991 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:46:19.866960 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.866837 4991 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:46:19.866960 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:19.866876 4991 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:46:19.891080 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.890995 4991 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="b48f5a4ff29408b4e28dda7e4b6704b1be8eb57707c6ebb9ca708056f6c6ebcc" exitCode=0 Oct 30 21:46:19.891080 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.891038 4991 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="debc53a084c45e1711152d4542e379fc44b804689b1498196f156852beba9a87" exitCode=0 Oct 30 21:46:19.891080 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.891046 4991 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="b64f9bd6bf2323ac8b0f2a231af9cfaf29f8ea768a1b4750eafa996d82830ab4" exitCode=0 Oct 30 21:46:19.891080 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.891053 4991 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="32e430384c42f8f98627988b1d4fd9942d5740648679194e56c9b5b666ed1aa9" exitCode=0 Oct 30 21:46:19.891080 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.891061 4991 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="bf9102c6954d81fc3611860cc61d08aa1295c4d6c31aa5fdbac120a18e280954" exitCode=0 Oct 30 21:46:19.891080 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.891068 4991 generic.go:356] "Generic (PLEG): container finished" podID="2a5249fa-084f-451b-8e94-9cbd497a8ea7" containerID="c8793d0efe53fd42b7d407508e6974a79e72c9634956751c25b5e5b09a95a099" exitCode=0 Oct 30 21:46:19.906304 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.906277 4991 generic.go:356] "Generic (PLEG): container finished" podID="fdf14b302ecfd0d0d1206ce2297bec21" containerID="9a1ef903ad4537dfc212d0dda17fed4d4a4c9e651874eb3e4ea06a078d15b4f6" exitCode=0 Oct 30 21:46:19.913232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.913212 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:46:19.913337 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.913240 4991 generic.go:356] "Generic (PLEG): container finished" podID="04efc6e4-19af-4935-a7c1-2dac67498e4c" containerID="c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b" exitCode=255 Oct 30 21:46:19.939704 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:19.939663 4991 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:46:19.943987 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.943970 4991 manager.go:324] Recovery completed Oct 30 21:46:19.961113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.961094 4991 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:46:19.962239 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.962222 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:46:19.962346 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.962251 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:46:19.962346 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.962261 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:46:19.967052 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:19.967030 4991 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:46:19.967413 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967395 4991 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:46:19.967467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967414 4991 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:46:19.967467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967436 4991 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:46:19.967682 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967665 4991 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:46:19.967749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967620 4991 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:46:19.967749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967720 4991 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:46:19.967839 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967754 4991 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:46:19.967839 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967763 4991 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:46:19.967839 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967810 4991 policy_none.go:50] "Start" Oct 30 21:46:19.967839 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967833 4991 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:46:19.968015 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.967850 4991 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:46:19.968133 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.968118 4991 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:46:19.968188 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.968135 4991 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:46:19.968188 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.968144 4991 policy_none.go:44] "Start" Oct 30 21:46:19.969193 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.969174 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:46:19.969278 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.969212 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:46:19.969278 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.969261 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:46:19.973271 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.973246 4991 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:46:19.978360 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.978052 4991 manager.go:341] "Starting Device Plugin manager" Oct 30 21:46:19.979747 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:19.979729 4991 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:46:19.979747 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.979748 4991 server.go:87] "Starting device plugin registration server" Oct 30 21:46:19.980768 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.980751 4991 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:46:19.980842 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.980774 4991 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:46:19.980893 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.980842 4991 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:46:19.981018 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.980977 4991 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:46:19.981119 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.981091 4991 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:46:19.981167 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.981119 4991 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:46:19.981280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.981190 4991 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:46:19.981010593Z","UUID":"8089acf5-7560-4a1b-b4ec-893441815593","Handler":null,"Name":"","Endpoint":""} Oct 30 21:46:19.982938 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.982606 4991 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:46:19.982938 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:19.982633 4991 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:46:19.983589 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:19.983570 4991 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:46:19.983741 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:19.983608 4991 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-17.us-west-2.compute.internal\" not found" Oct 30 21:46:20.013884 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.013831 4991 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:46:20.014868 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.014849 4991 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:46:20.014927 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.014904 4991 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:46:20.014927 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.014914 4991 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:46:20.015646 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.015630 4991 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:46:20.081729 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.081705 4991 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:46:20.082743 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.082726 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:46:20.082808 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.082756 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:46:20.082808 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.082771 4991 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-17.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:46:20.082864 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.082823 4991 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.167744 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.167658 4991 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal"] Oct 30 21:46:20.168424 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.168392 4991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.168571 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.168555 4991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.169026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.168983 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"fdf14b302ecfd0d0d1206ce2297bec21","Type":"ContainerStarted","Data":"36b378df4cdc8bde691544927d8a7f939cdaa05f53238fb924f3b12b0564167d"} Oct 30 21:46:20.169082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.169030 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"fdf14b302ecfd0d0d1206ce2297bec21","Type":"ContainerDied","Data":"9a1ef903ad4537dfc212d0dda17fed4d4a4c9e651874eb3e4ea06a078d15b4f6"} Oct 30 21:46:20.169082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.169044 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"fdf14b302ecfd0d0d1206ce2297bec21","Type":"ContainerStarted","Data":"a84bc6d1fa3eab6df9b32ecf2230e86c962dfb59c8044483740a2009387dd05a"} Oct 30 21:46:20.169082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.169072 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"44d4a0aec485b9ad44110c51253a50d0","Type":"ContainerStarted","Data":"d7d72df3f2394312417c189db556e7a76a43dc9b2ed9283b8385819ae1743523"} Oct 30 21:46:20.169220 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.169084 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" event={"ID":"44d4a0aec485b9ad44110c51253a50d0","Type":"ContainerStarted","Data":"05d1587f9284f1a39eae6c1d904770c6523b60ae1c5ce1a15c6907cfff06b924"} Oct 30 21:46:20.242597 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.242461 4991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:46:20.242597 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:20.242516 4991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.249268 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.249243 4991 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.249423 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.249309 4991 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.255225 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.255206 4991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:46:20.255338 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:20.255246 4991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.318147 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.318126 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44d4a0aec485b9ad44110c51253a50d0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"44d4a0aec485b9ad44110c51253a50d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.318244 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.318154 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.318244 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.318176 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.418477 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.418398 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.418477 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.418455 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.418654 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.418482 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44d4a0aec485b9ad44110c51253a50d0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"44d4a0aec485b9ad44110c51253a50d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.418654 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.418546 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44d4a0aec485b9ad44110c51253a50d0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"44d4a0aec485b9ad44110c51253a50d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.418654 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.418602 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.418793 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.418656 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf14b302ecfd0d0d1206ce2297bec21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" (UID: \"fdf14b302ecfd0d0d1206ce2297bec21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.764362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.764338 4991 apiserver.go:52] "Watching apiserver" Oct 30 21:46:20.913361 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.913314 4991 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:46:20.914010 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.913974 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-r2dj5","openshift-cluster-node-tuning-operator/tuned-vwfkb","openshift-image-registry/image-registry-58d68ffbc4-r5mnd","openshift-network-diagnostics/network-check-target-tqcc7","openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c","kube-system/kas-connection-checker-685895c7c9-dll8d","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d","openshift-multus/network-metrics-daemon-wbmj9","openshift-ingress/router-default-5c8c9db44f-llgx4","openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb","openshift-image-registry/image-registry-5b65fb6c59-872xp","openshift-image-registry/node-ca-xm8l9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg","openshift-network-console/networking-console-plugin-86c478bb88-j66r8","openshift-insights/insights-operator-7c4986d449-tc7w4","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw","openshift-multus/multus-746ws","kube-system/global-pull-secret-syncer-4zhvv","openshift-console-operator/console-operator-7cd56b8dbc-jqbm4","openshift-dns/node-resolver-g66br","openshift-ovn-kubernetes/ovnkube-node-lgx8f","kube-system/kas-connection-checker-685895c7c9-2jxfz","openshift-ingress-canary/ingress-canary-qdqzz","openshift-insights/insights-runtime-extractor-bhs7l","openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr","kube-system/kas-connection-checker-685895c7c9-p2dr2","kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p","openshift-multus/multus-additional-cni-plugins-vznvs","openshift-insights/periodic-gathering-p8p6k-4z6rv","openshift-service-ca/service-ca-5fffc97bdf-7hrj8","openshift-network-operator/iptables-alerter-fmwmz","openshift-dns/dns-default-8fwrp"] Oct 30 21:46:20.915267 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.915244 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:20.915994 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.915974 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:20.916391 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.916371 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:20.917101 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.917081 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:20.918811 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.918794 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:20.919621 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.919607 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:20.926995 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.926978 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:20.928300 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.928280 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:20.929218 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.929199 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:20.931952 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.931926 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qdqzz_2da4fada-55a5-42ad-b873-46f86929c5d0/serve-healthcheck-canary/0.log" Oct 30 21:46:20.933623 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.933600 4991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.934122 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.934104 4991 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:20.940579 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.940560 4991 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:46:20.969504 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.969480 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:46:20.969674 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.969659 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:46:20.969783 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.969768 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:46:20.969834 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.969787 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:46:20.970752 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.970738 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:46:20.970884 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.970868 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:46:20.970944 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.970930 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:46:20.970987 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.970964 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:46:20.970987 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.970974 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:46:20.971065 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:20.970998 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.017024 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.016961 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:46:21.017119 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017033 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:46:21.017119 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017039 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:46:21.017119 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017106 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:46:21.017269 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017157 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:46:21.017269 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017123 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:46:21.017269 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017216 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:46:21.017269 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017257 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:46:21.017269 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017264 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.017269 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017260 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.017524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017300 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:46:21.017524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017301 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:46:21.017524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017365 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:46:21.017524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017367 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:46:21.017524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017426 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5wdm\"" Oct 30 21:46:21.017524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.017481 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mkvf\"" Oct 30 21:46:21.018536 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018519 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:46:21.018665 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018573 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:46:21.018665 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018639 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:46:21.018665 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018654 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\"" Oct 30 21:46:21.018796 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018667 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5wwct\"" Oct 30 21:46:21.018796 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018782 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-879p5\"" Oct 30 21:46:21.018887 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018837 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:46:21.018943 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018892 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hs26f\"" Oct 30 21:46:21.018991 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018944 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:46:21.018991 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.018951 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tmn85\"" Oct 30 21:46:21.019080 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019047 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rn9hr\"" Oct 30 21:46:21.019238 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019222 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:46:21.019359 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019305 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:46:21.019359 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019336 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019297 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kmf25\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019289 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bg2hw\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019313 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kjn6x\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019403 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019296 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xlfcr\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019352 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tjx5c\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019378 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019447 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9dfpb\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019343 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5mqqn\"" Oct 30 21:46:21.019510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019379 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:46:21.020038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019560 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pvjtk\"" Oct 30 21:46:21.020038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019594 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqqg2\"" Oct 30 21:46:21.020038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019676 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fd5qx\"" Oct 30 21:46:21.020038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019726 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g8m85\"" Oct 30 21:46:21.020038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019813 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pz9m9\"" Oct 30 21:46:21.020038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.019889 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:46:21.021515 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021497 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-etc-kubernetes\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.021638 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021524 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-socket-dir-parent\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.021638 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021543 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.021638 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021560 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-stats-auth\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.021638 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021621 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2xf\" (UniqueName: \"kubernetes.io/projected/8743ad27-5d90-4b7f-8012-6faf21548f97-kube-api-access-7s2xf\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.021840 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021651 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.021840 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021680 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-lib-modules\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.021840 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021707 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4qt\" (UniqueName: \"kubernetes.io/projected/5af23954-d911-4664-872d-9ae2e67ecfa0-kube-api-access-cw4qt\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.021840 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021740 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.022082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021928 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:21.022082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.021957 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-stats-auth\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.022082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022003 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95nz\" (UniqueName: \"kubernetes.io/projected/f793981d-e86b-4866-920e-aa96d4de9e30-kube-api-access-k95nz\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:21.022240 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022096 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.022240 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022127 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b9b148-13a4-40f3-9113-2877f5ca9574-host\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:46:21.022240 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022157 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3afb35-bfde-45ec-a972-78a6e7461d9d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:21.022240 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022201 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.022240 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022227 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-registration-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.022526 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022245 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.022526 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022281 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvtf\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.022526 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022349 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a07200b0-a7f6-4c5a-9197-b73779a06c1f-serving-cert\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.022526 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022412 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.022526 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022450 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.022749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022545 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxtvm\" (UniqueName: \"kubernetes.io/projected/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a-kube-api-access-nxtvm\") pod \"kas-connection-checker-685895c7c9-2jxfz\" (UID: \"6ffd4edc-ce40-4a48-b398-c2d907a7cd0a\") " pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" Oct 30 21:46:21.022749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022566 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a07200b0-a7f6-4c5a-9197-b73779a06c1f-serving-cert\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.022749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022588 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-kubelet-config\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.022749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022674 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-systemd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.022749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022702 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022823 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-cni-binary-copy\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022877 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022906 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8743ad27-5d90-4b7f-8012-6faf21548f97-tmp-dir\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022924 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-sys-fs\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022939 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-host\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.022956 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67tn\" (UniqueName: \"kubernetes.io/projected/04efc6e4-19af-4935-a7c1-2dac67498e4c-kube-api-access-q67tn\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023000 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023021 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023036 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023035 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023043 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8743ad27-5d90-4b7f-8012-6faf21548f97-tmp-dir\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.023071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023061 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qrq\" (UniqueName: \"kubernetes.io/projected/dbdaddf0-875f-46f2-8418-62377b63a086-kube-api-access-22qrq\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023187 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-kubelet\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023192 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023228 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-multus\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023281 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-multus-certs\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023308 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023353 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e021343-49b5-4868-a1f8-3a1bd283c42c-crio-socket\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023378 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023396 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58vn\" (UniqueName: \"kubernetes.io/projected/16867cf6-0526-47b2-866e-5dfc13f04869-kube-api-access-l58vn\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023411 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-config\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023427 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-agent-certs\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023446 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023544 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023568 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-modprobe-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023591 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023591 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.023637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023618 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f793981d-e86b-4866-920e-aa96d4de9e30-signing-cabundle\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023655 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqdn\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023697 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-log-socket\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023765 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023820 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45q9q\" (UniqueName: \"kubernetes.io/projected/c6eb7759-f63d-4817-9128-adcecdceb0d7-kube-api-access-45q9q\") pod \"volume-data-source-validator-58c9f5f46d-h6cxw\" (UID: \"c6eb7759-f63d-4817-9128-adcecdceb0d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023841 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-agent-certs\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023850 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023885 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023915 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3280f18-2d95-4303-8f34-5a2fe91671e6-original-pull-secret\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023927 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023957 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-env-overrides\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.023977 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024006 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-hostroot\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024030 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7xlh\" (UniqueName: \"kubernetes.io/projected/0abc3e92-4ede-4421-ab8d-91477bd6b85e-kube-api-access-r7xlh\") pod \"kas-connection-checker-685895c7c9-p2dr2\" (UID: \"0abc3e92-4ede-4421-ab8d-91477bd6b85e\") " pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024047 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-config\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024082 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04efc6e4-19af-4935-a7c1-2dac67498e4c-serving-cert\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024166 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-systemd-units\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024201 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sz9x\" (UniqueName: \"kubernetes.io/projected/2da4fada-55a5-42ad-b873-46f86929c5d0-kube-api-access-5sz9x\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024200 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024236 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-var-lib-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024264 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkccl\" (UniqueName: \"kubernetes.io/projected/f008c236-2573-46f0-bbdd-f10daa5d08bf-kube-api-access-lkccl\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024291 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqg42\" (UniqueName: \"kubernetes.io/projected/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-kube-api-access-cqg42\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024335 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqd5\" (UniqueName: \"kubernetes.io/projected/d6594a6f-9c78-40bf-99b7-fb21cd031157-kube-api-access-rtqd5\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:21.024385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024383 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxspb\" (UniqueName: \"kubernetes.io/projected/3ac87e61-9d62-4657-90a3-2bf7e4877f85-kube-api-access-fxspb\") pod \"migrator-66df5d496b-5vnv5\" (UID: \"3ac87e61-9d62-4657-90a3-2bf7e4877f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024421 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-conf\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024450 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16867cf6-0526-47b2-866e-5dfc13f04869-hosts-file\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024479 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovn-node-metrics-cert\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024516 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024548 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-system-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024583 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04efc6e4-19af-4935-a7c1-2dac67498e4c-serving-cert\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024579 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq4g\" (UniqueName: \"kubernetes.io/projected/a547a8cd-8909-475e-a03d-15b409877777-kube-api-access-7nq4g\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024621 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024650 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtl9\" (UniqueName: \"kubernetes.io/projected/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kube-api-access-vjtl9\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024676 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024709 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024762 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-kubelet\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024773 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovn-node-metrics-cert\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024909 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47qr\" (UniqueName: \"kubernetes.io/projected/12b9b148-13a4-40f3-9113-2877f5ca9574-kube-api-access-j47qr\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024942 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-default-certificate\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.024968 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3afb35-bfde-45ec-a972-78a6e7461d9d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025012 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6594a6f-9c78-40bf-99b7-fb21cd031157-serving-cert\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025041 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6594a6f-9c78-40bf-99b7-fb21cd031157-config\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025049 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025068 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82np\" (UniqueName: \"kubernetes.io/projected/552b3503-9502-4ac7-af98-0acbc947254c-kube-api-access-d82np\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025098 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-etc-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025127 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-node-log\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025152 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-bin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025183 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntnrc\" (UniqueName: \"kubernetes.io/projected/2a5249fa-084f-451b-8e94-9cbd497a8ea7-kube-api-access-ntnrc\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025198 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3afb35-bfde-45ec-a972-78a6e7461d9d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025209 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025230 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-default-certificate\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025278 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-device-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025303 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysconfig\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025358 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxh4\" (UniqueName: \"kubernetes.io/projected/6b4c949e-cff6-4abf-b3ff-f85046f76784-kube-api-access-2pxh4\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025380 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6594a6f-9c78-40bf-99b7-fb21cd031157-serving-cert\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025386 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025436 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-k8s-cni-cncf-io\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.025448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025467 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnkh\" (UniqueName: \"kubernetes.io/projected/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-api-access-2fnkh\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025513 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-sys\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025591 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025608 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025644 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025662 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-netd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025680 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025707 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-snapshots\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025751 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cnibin\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025831 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025830 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025855 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-snapshots\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025859 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025872 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025908 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025933 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025962 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-systemd\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.025984 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-bin\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026017 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-cnibin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026097 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026123 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcqn\" (UniqueName: \"kubernetes.io/projected/a07200b0-a7f6-4c5a-9197-b73779a06c1f-kube-api-access-rpcqn\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026224 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwfx\" (UniqueName: \"kubernetes.io/projected/4f3afb35-bfde-45ec-a972-78a6e7461d9d-kube-api-access-thwfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026241 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07200b0-a7f6-4c5a-9197-b73779a06c1f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026245 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-socket-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026297 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-etc-tuned\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026365 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-netns\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026399 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-etc-tuned\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026413 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-daemon-config\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026460 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-tmp\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026481 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8743ad27-5d90-4b7f-8012-6faf21548f97-config-volume\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026497 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e021343-49b5-4868-a1f8-3a1bd283c42c-data-volume\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026522 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7mx\" (UniqueName: \"kubernetes.io/projected/177b7d0a-d868-497a-b648-23228667c743-kube-api-access-7z7mx\") pod \"kas-connection-checker-685895c7c9-dll8d\" (UID: \"177b7d0a-d868-497a-b648-23228667c743\") " pod="kube-system/kas-connection-checker-685895c7c9-dll8d" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026554 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-netns\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026568 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07200b0-a7f6-4c5a-9197-b73779a06c1f-tmp\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026578 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e021343-49b5-4868-a1f8-3a1bd283c42c-data-volume\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026663 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da4fada-55a5-42ad-b873-46f86929c5d0-cert\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026694 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026717 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026760 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c494bc9-5908-48bc-9825-2669ad4196c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026805 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-slash\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026839 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026839 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026865 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-script-lib\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026870 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2da4fada-55a5-42ad-b873-46f86929c5d0-cert\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026888 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-konnectivity-ca\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026906 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-conf-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026921 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdh8\" (UniqueName: \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026941 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/552b3503-9502-4ac7-af98-0acbc947254c-iptables-alerter-script\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.026975 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.027026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027051 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027075 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jtl\" (UniqueName: \"kubernetes.io/projected/50406e5f-7f63-49e5-aca1-d572b155292a-kube-api-access-f5jtl\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027092 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027109 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027128 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027132 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b9b148-13a4-40f3-9113-2877f5ca9574-serviceca\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027157 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-run\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027185 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl44l\" (UniqueName: \"kubernetes.io/projected/74d26ea6-81b0-4ff7-acef-cd28751f4a78-kube-api-access-sl44l\") pod \"migrator-66df5d496b-ssjmb\" (UID: \"74d26ea6-81b0-4ff7-acef-cd28751f4a78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027210 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f793981d-e86b-4866-920e-aa96d4de9e30-signing-key\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027236 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd5v\" (UniqueName: \"kubernetes.io/projected/7e37f363-3bfe-4f18-a5ee-f60f849499d1-kube-api-access-7bd5v\") pod \"network-check-source-6f78d59f99-qzvdr\" (UID: \"7e37f363-3bfe-4f18-a5ee-f60f849499d1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027236 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027283 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027304 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b9b148-13a4-40f3-9113-2877f5ca9574-serviceca\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027418 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16867cf6-0526-47b2-866e-5dfc13f04869-tmp-dir\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027440 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-ovn\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027456 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027475 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f793981d-e86b-4866-920e-aa96d4de9e30-signing-key\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027495 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027537 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16867cf6-0526-47b2-866e-5dfc13f04869-tmp-dir\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027540 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-dbus\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027575 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-kubernetes\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027608 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-var-lib-kubelet\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027638 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-tmp\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027672 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027710 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50406e5f-7f63-49e5-aca1-d572b155292a-tmp\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.029069 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.027718 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552b3503-9502-4ac7-af98-0acbc947254c-host-slash\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:46:21.033919 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.033902 4991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:46:21.033966 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.033948 4991 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:46:21.033996 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.033986 4991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:21.034032 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.033951 4991 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-17.us-west-2.compute.internal" Oct 30 21:46:21.068994 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.068968 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.069093 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069085 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.069156 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069103 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:46:21.069362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069350 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.069416 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069392 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.069416 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069403 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.069542 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069426 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.069632 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069557 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.069632 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069574 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.069632 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069617 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.069632 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.069629 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.072293 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.072275 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.073551 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.073534 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-cni-binary-copy\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.073624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.073576 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:46:21.073660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.073632 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.075183 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.075153 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:46:21.075263 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.075198 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.075389 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.075376 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.075732 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.075719 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.077115 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077101 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.077232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077165 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.077232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077184 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:46:21.077232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077200 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:46:21.077232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077212 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.077544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077375 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.077544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077376 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:46:21.077544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077390 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.077544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077489 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:46:21.077544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077526 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:46:21.077544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077532 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.077544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077545 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077543 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077592 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077600 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077594 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077624 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077602 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077578 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077633 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077638 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077643 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077643 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077651 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077661 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077738 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077809 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077848 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:46:21.078053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.077896 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.078584 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.078272 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:46:21.082248 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.078964 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:46:21.082248 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.079189 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:46:21.082248 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.079607 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:46:21.082248 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.080696 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qrq\" (UniqueName: \"kubernetes.io/projected/dbdaddf0-875f-46f2-8418-62377b63a086-kube-api-access-22qrq\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:21.082673 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.082656 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3afb35-bfde-45ec-a972-78a6e7461d9d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:21.082758 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.082741 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.084532 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.084507 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-env-overrides\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.084624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.084516 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-config\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.084624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.084558 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-config\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.084706 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.084626 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f793981d-e86b-4866-920e-aa96d4de9e30-signing-cabundle\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:21.085815 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.085799 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6594a6f-9c78-40bf-99b7-fb21cd031157-config\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:21.085880 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.085796 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:21.086960 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.086941 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c494bc9-5908-48bc-9825-2669ad4196c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:21.086960 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.086951 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8743ad27-5d90-4b7f-8012-6faf21548f97-config-volume\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.087048 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.086962 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-daemon-config\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.087078 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.087057 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5249fa-084f-451b-8e94-9cbd497a8ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.087078 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.087071 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fad6427c-0014-4c83-9ff9-ce1d9ba0aa55-konnectivity-ca\") pod \"konnectivity-agent-r2dj5\" (UID: \"fad6427c-0014-4c83-9ff9-ce1d9ba0aa55\") " pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:46:21.087251 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.087234 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f008c236-2573-46f0-bbdd-f10daa5d08bf-ovnkube-script-lib\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.087369 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.087350 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/552b3503-9502-4ac7-af98-0acbc947254c-iptables-alerter-script\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:46:21.087544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.087529 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04efc6e4-19af-4935-a7c1-2dac67498e4c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.087893 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.087878 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4qt\" (UniqueName: \"kubernetes.io/projected/5af23954-d911-4664-872d-9ae2e67ecfa0-kube-api-access-cw4qt\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.088139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.088123 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzk64\" (UniqueName: \"kubernetes.io/projected/573e461a-8123-4e53-9134-80d6eb9a3e71-kube-api-access-dzk64\") pod \"network-check-target-tqcc7\" (UID: \"573e461a-8123-4e53-9134-80d6eb9a3e71\") " pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:21.098386 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.098368 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnkh\" (UniqueName: \"kubernetes.io/projected/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-api-access-2fnkh\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.103202 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.103183 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqd5\" (UniqueName: \"kubernetes.io/projected/d6594a6f-9c78-40bf-99b7-fb21cd031157-kube-api-access-rtqd5\") pod \"service-ca-operator-544d749c8c-6kc9p\" (UID: \"d6594a6f-9c78-40bf-99b7-fb21cd031157\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" Oct 30 21:46:21.103423 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.103408 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq4g\" (UniqueName: \"kubernetes.io/projected/a547a8cd-8909-475e-a03d-15b409877777-kube-api-access-7nq4g\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.105832 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.105816 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7xlh\" (UniqueName: \"kubernetes.io/projected/0abc3e92-4ede-4421-ab8d-91477bd6b85e-kube-api-access-r7xlh\") pod \"kas-connection-checker-685895c7c9-p2dr2\" (UID: \"0abc3e92-4ede-4421-ab8d-91477bd6b85e\") " pod="kube-system/kas-connection-checker-685895c7c9-p2dr2" Oct 30 21:46:21.108936 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.108914 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7mx\" (UniqueName: \"kubernetes.io/projected/177b7d0a-d868-497a-b648-23228667c743-kube-api-access-7z7mx\") pod \"kas-connection-checker-685895c7c9-dll8d\" (UID: \"177b7d0a-d868-497a-b648-23228667c743\") " pod="kube-system/kas-connection-checker-685895c7c9-dll8d" Oct 30 21:46:21.109018 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.108998 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntnrc\" (UniqueName: \"kubernetes.io/projected/2a5249fa-084f-451b-8e94-9cbd497a8ea7-kube-api-access-ntnrc\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.109074 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.109062 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcqn\" (UniqueName: \"kubernetes.io/projected/a07200b0-a7f6-4c5a-9197-b73779a06c1f-kube-api-access-rpcqn\") pod \"insights-operator-7c4986d449-tc7w4\" (UID: \"a07200b0-a7f6-4c5a-9197-b73779a06c1f\") " pod="openshift-insights/insights-operator-7c4986d449-tc7w4" Oct 30 21:46:21.109230 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.109214 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxh4\" (UniqueName: \"kubernetes.io/projected/6b4c949e-cff6-4abf-b3ff-f85046f76784-kube-api-access-2pxh4\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:21.111627 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.111602 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkccl\" (UniqueName: \"kubernetes.io/projected/f008c236-2573-46f0-bbdd-f10daa5d08bf-kube-api-access-lkccl\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.111710 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.111630 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jtl\" (UniqueName: \"kubernetes.io/projected/50406e5f-7f63-49e5-aca1-d572b155292a-kube-api-access-f5jtl\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.111710 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.111695 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67tn\" (UniqueName: \"kubernetes.io/projected/04efc6e4-19af-4935-a7c1-2dac67498e4c-kube-api-access-q67tn\") pod \"console-operator-7cd56b8dbc-jqbm4\" (UID: \"04efc6e4-19af-4935-a7c1-2dac67498e4c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.111792 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.111774 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82np\" (UniqueName: \"kubernetes.io/projected/552b3503-9502-4ac7-af98-0acbc947254c-kube-api-access-d82np\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:46:21.111886 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.111872 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45q9q\" (UniqueName: \"kubernetes.io/projected/c6eb7759-f63d-4817-9128-adcecdceb0d7-kube-api-access-45q9q\") pod \"volume-data-source-validator-58c9f5f46d-h6cxw\" (UID: \"c6eb7759-f63d-4817-9128-adcecdceb0d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h6cxw" Oct 30 21:46:21.111929 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.111875 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqdn\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.114116 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.114097 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdh8\" (UniqueName: \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\") pod \"periodic-gathering-p8p6k-4z6rv\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:46:21.114211 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.114166 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvtf\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.114528 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.114510 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95nz\" (UniqueName: \"kubernetes.io/projected/f793981d-e86b-4866-920e-aa96d4de9e30-kube-api-access-k95nz\") pod \"service-ca-5fffc97bdf-7hrj8\" (UID: \"f793981d-e86b-4866-920e-aa96d4de9e30\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7hrj8" Oct 30 21:46:21.115936 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.115912 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxtvm\" (UniqueName: \"kubernetes.io/projected/6ffd4edc-ce40-4a48-b398-c2d907a7cd0a-kube-api-access-nxtvm\") pod \"kas-connection-checker-685895c7c9-2jxfz\" (UID: \"6ffd4edc-ce40-4a48-b398-c2d907a7cd0a\") " pod="kube-system/kas-connection-checker-685895c7c9-2jxfz" Oct 30 21:46:21.116010 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.115998 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxspb\" (UniqueName: \"kubernetes.io/projected/3ac87e61-9d62-4657-90a3-2bf7e4877f85-kube-api-access-fxspb\") pod \"migrator-66df5d496b-5vnv5\" (UID: \"3ac87e61-9d62-4657-90a3-2bf7e4877f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" Oct 30 21:46:21.116602 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.116586 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47qr\" (UniqueName: \"kubernetes.io/projected/12b9b148-13a4-40f3-9113-2877f5ca9574-kube-api-access-j47qr\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:46:21.117310 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.117296 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sz9x\" (UniqueName: \"kubernetes.io/projected/2da4fada-55a5-42ad-b873-46f86929c5d0-kube-api-access-5sz9x\") pod \"ingress-canary-qdqzz\" (UID: \"2da4fada-55a5-42ad-b873-46f86929c5d0\") " pod="openshift-ingress-canary/ingress-canary-qdqzz" Oct 30 21:46:21.117500 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.117483 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwfx\" (UniqueName: \"kubernetes.io/projected/4f3afb35-bfde-45ec-a972-78a6e7461d9d-kube-api-access-thwfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c\" (UID: \"4f3afb35-bfde-45ec-a972-78a6e7461d9d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" Oct 30 21:46:21.118083 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.118069 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtl9\" (UniqueName: \"kubernetes.io/projected/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kube-api-access-vjtl9\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.118489 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.118474 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.119501 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.119485 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqg42\" (UniqueName: \"kubernetes.io/projected/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-kube-api-access-cqg42\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:21.122602 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.122587 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58vn\" (UniqueName: \"kubernetes.io/projected/16867cf6-0526-47b2-866e-5dfc13f04869-kube-api-access-l58vn\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:46:21.124640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.124625 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2xf\" (UniqueName: \"kubernetes.io/projected/8743ad27-5d90-4b7f-8012-6faf21548f97-kube-api-access-7s2xf\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.126413 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.126398 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl44l\" (UniqueName: \"kubernetes.io/projected/74d26ea6-81b0-4ff7-acef-cd28751f4a78-kube-api-access-sl44l\") pod \"migrator-66df5d496b-ssjmb\" (UID: \"74d26ea6-81b0-4ff7-acef-cd28751f4a78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" Oct 30 21:46:21.126877 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.126862 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd5v\" (UniqueName: \"kubernetes.io/projected/7e37f363-3bfe-4f18-a5ee-f60f849499d1-kube-api-access-7bd5v\") pod \"network-check-source-6f78d59f99-qzvdr\" (UID: \"7e37f363-3bfe-4f18-a5ee-f60f849499d1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qzvdr" Oct 30 21:46:21.127582 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.127567 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.128513 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128499 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-multus\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.128548 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128527 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-multus-certs\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.128548 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128543 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.128621 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128556 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e021343-49b5-4868-a1f8-3a1bd283c42c-crio-socket\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.128621 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128586 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.128621 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128607 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-modprobe-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.128621 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128607 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-multus\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.128621 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128598 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-multus-certs\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128630 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128647 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128653 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e021343-49b5-4868-a1f8-3a1bd283c42c-crio-socket\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128671 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128697 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-log-socket\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128719 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-modprobe-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128722 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128737 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-log-socket\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128753 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-d\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128755 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:21.128795 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.128796 4991 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.128822 4991 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128846 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128797 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.128868 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.62885461 +0000 UTC m=+1.928264503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.128898 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.628885392 +0000 UTC m=+1.928295253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128926 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-hostroot\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128949 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-systemd-units\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128972 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-var-lib-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.128998 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-conf\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129017 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-hostroot\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129025 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-systemd-units\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129018 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16867cf6-0526-47b2-866e-5dfc13f04869-hosts-file\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129056 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-var-lib-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129064 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-system-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129074 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16867cf6-0526-47b2-866e-5dfc13f04869-hosts-file\") pod \"node-resolver-g66br\" (UID: \"16867cf6-0526-47b2-866e-5dfc13f04869\") " pod="openshift-dns/node-resolver-g66br" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129109 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-system-cni-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129106 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129136 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129151 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-kubelet\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129121 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysctl-conf\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129159 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129171 4991 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129181 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-etc-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129202 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-node-log\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129209 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-kubelet\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129215 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129228 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.629211959 +0000 UTC m=+1.928621834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129233 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129234 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-etc-openvswitch\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129241 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-node-log\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129250 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-bin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129277 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-var-lib-cni-bin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129284 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.62927142 +0000 UTC m=+1.928681300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129296 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129312 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-device-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129347 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysconfig\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129371 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-k8s-cni-cncf-io\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129385 4991 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129397 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-sys\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129401 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-sysconfig\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129375 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-device-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129420 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-k8s-cni-cncf-io\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129433 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.629420025 +0000 UTC m=+1.928829897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129421 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-netd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129444 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-sys\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129453 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-netd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129457 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cnibin\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129479 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129514 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-cnibin\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129524 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129544 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-systemd\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129549 4991 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129558 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-bin\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129572 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-cnibin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129593 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.629583579 +0000 UTC m=+1.928993436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129605 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-socket-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129619 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-cnibin\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129625 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-systemd\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129627 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-netns\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129631 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-cni-bin\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129658 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-netns\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129671 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-socket-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129663 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-netns\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129703 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129725 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-slash\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129743 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-host-run-netns\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129757 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-conf-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129782 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-slash\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129790 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.629776338 +0000 UTC m=+1.929186218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : configmap references non-existent config key: service-ca.crt Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129804 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-conf-dir\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129814 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-run\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129836 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-run\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129846 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-ovn\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129871 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129900 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129924 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-ovn\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129934 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-dbus\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.129928 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.129998 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130004 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-dbus\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.129998 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.130013 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130010 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-kubernetes\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130053 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-etc-kubernetes\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.130074 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.630053318 +0000 UTC m=+1.929463223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130094 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-var-lib-kubelet\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130106 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e021343-49b5-4868-a1f8-3a1bd283c42c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130125 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552b3503-9502-4ac7-af98-0acbc947254c-host-slash\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130152 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-etc-kubernetes\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130166 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-socket-dir-parent\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130111 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-var-lib-kubelet\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130184 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130205 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-lib-modules\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130216 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-multus-socket-dir-parent\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130182 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552b3503-9502-4ac7-af98-0acbc947254c-host-slash\") pod \"iptables-alerter-fmwmz\" (UID: \"552b3503-9502-4ac7-af98-0acbc947254c\") " pod="openshift-network-operator/iptables-alerter-fmwmz" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130230 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130223 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130205 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af23954-d911-4664-872d-9ae2e67ecfa0-etc-kubernetes\") pod \"multus-746ws\" (UID: \"5af23954-d911-4664-872d-9ae2e67ecfa0\") " pod="openshift-multus/multus-746ws" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130265 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-lib-modules\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130271 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b9b148-13a4-40f3-9113-2877f5ca9574-host\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130275 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5249fa-084f-451b-8e94-9cbd497a8ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-vznvs\" (UID: \"2a5249fa-084f-451b-8e94-9cbd497a8ea7\") " pod="openshift-multus/multus-additional-cni-plugins-vznvs" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130293 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b9b148-13a4-40f3-9113-2877f5ca9574-host\") pod \"node-ca-xm8l9\" (UID: \"12b9b148-13a4-40f3-9113-2877f5ca9574\") " pod="openshift-image-registry/node-ca-xm8l9" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130297 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130337 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-registration-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130351 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130365 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-kubelet-config\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130394 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-systemd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130395 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-registration-dir\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130422 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-sys-fs\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130452 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-host\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130462 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-run-systemd\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130424 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3280f18-2d95-4303-8f34-5a2fe91671e6-kubelet-config\") pod \"global-pull-secret-syncer-4zhvv\" (UID: \"d3280f18-2d95-4303-8f34-5a2fe91671e6\") " pod="kube-system/global-pull-secret-syncer-4zhvv" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130478 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d9b2124f-2877-4c7b-8fc1-75d3bda8c459-sys-fs\") pod \"aws-ebs-csi-driver-node-bvkmg\" (UID: \"d9b2124f-2877-4c7b-8fc1-75d3bda8c459\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvkmg" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130475 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130515 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130519 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130527 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50406e5f-7f63-49e5-aca1-d572b155292a-host\") pod \"tuned-vwfkb\" (UID: \"50406e5f-7f63-49e5-aca1-d572b155292a\") " pod="openshift-cluster-node-tuning-operator/tuned-vwfkb" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130553 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.130573 4991 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130580 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-kubelet\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.130604 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.630592526 +0000 UTC m=+1.930002389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.130646 4991 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.130681 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs podName:dbdaddf0-875f-46f2-8418-62377b63a086 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.630670418 +0000 UTC m=+1.930080313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs") pod "network-metrics-daemon-wbmj9" (UID: "dbdaddf0-875f-46f2-8418-62377b63a086") : secret "metrics-daemon-secret" not found Oct 30 21:46:21.132139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.130649 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f008c236-2573-46f0-bbdd-f10daa5d08bf-host-kubelet\") pod \"ovnkube-node-lgx8f\" (UID: \"f008c236-2573-46f0-bbdd-f10daa5d08bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:21.218381 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.218364 4991 scope.go:122] "RemoveContainer" containerID="c58359654bd303fc5f5112574f43beab8dcf3c67fe9c5e7505d05297cecbb54b" Oct 30 21:46:21.233102 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.233086 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:46:21.472342 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.472300 4991 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:21.531971 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.531942 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c_4f3afb35-bfde-45ec-a972-78a6e7461d9d/kube-storage-version-migrator-operator/0.log" Oct 30 21:46:21.634759 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.634702 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:21.634759 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.634730 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:21.634759 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.634755 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:21.634930 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.634815 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:21.634930 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.634864 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:21.634930 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.634886 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:21.634930 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.634914 4991 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.634972 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.634956742 +0000 UTC m=+2.934366603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.634976 4991 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.634917 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.634977 4991 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635031 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.635009285 +0000 UTC m=+2.934419168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.635046 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635048 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635054 4991 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635051 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.635040998 +0000 UTC m=+2.934450856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635085 4991 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635067 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635111 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.635095904 +0000 UTC m=+2.934505779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:21.635121 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635131 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.635120845 +0000 UTC m=+2.934530706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.635153 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.635155 4991 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.635207 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635256 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.635239423 +0000 UTC m=+2.934649286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635313 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635348 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635387 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.63537352 +0000 UTC m=+2.934783391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635477 4991 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:21.635514 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:22.635503271 +0000 UTC m=+2.934913146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:21.635688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.635667 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a547a8cd-8909-475e-a03d-15b409877777-service-ca-bundle\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:21.637601 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.637581 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdaddf0-875f-46f2-8418-62377b63a086-metrics-certs\") pod \"network-metrics-daemon-wbmj9\" (UID: \"dbdaddf0-875f-46f2-8418-62377b63a086\") " pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:21.820501 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.820481 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbmj9" Oct 30 21:46:21.938026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.937995 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:46:21.939048 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:21.939020 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" event={"ID":"04efc6e4-19af-4935-a7c1-2dac67498e4c","Type":"ContainerStarted","Data":"eb6bea4234a25565b4e561b3ef75bc508e0d106a376e62ec2d613fe742095af3"} Oct 30 21:46:22.210771 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.210745 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbmj9"] Oct 30 21:46:22.217309 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:22.217272 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdaddf0_875f_46f2_8418_62377b63a086.slice/crio-fc6a7afbd1015b47dce34ff2b0fa0a995d4e751aef0395151d2332e4e5cdea65 WatchSource:0}: Error finding container fc6a7afbd1015b47dce34ff2b0fa0a995d4e751aef0395151d2332e4e5cdea65: Status 404 returned error can't find the container with id fc6a7afbd1015b47dce34ff2b0fa0a995d4e751aef0395151d2332e4e5cdea65 Oct 30 21:46:22.641268 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641239 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:22.641448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641286 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:22.641448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641314 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:22.641448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641396 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:22.641448 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641408 4991 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:22.641448 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641427 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:22.641448 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641448 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641466 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641448171 +0000 UTC m=+4.940858050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641428 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641431 4991 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641482 4991 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641496 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641480551 +0000 UTC m=+4.940890436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641482 4991 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641520 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls podName:8743ad27-5d90-4b7f-8012-6faf21548f97 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641508134 +0000 UTC m=+4.940917995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls") pod "dns-default-8fwrp" (UID: "8743ad27-5d90-4b7f-8012-6faf21548f97") : secret "dns-default-metrics-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641536 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls podName:67d09fe6-fb0a-43cc-8b20-98cdb22eea2a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641530116 +0000 UTC m=+4.940939973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dvh4d" (UID: "67d09fe6-fb0a-43cc-8b20-98cdb22eea2a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641553 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641544988 +0000 UTC m=+4.940954851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641596 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641617 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:22.641680 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.641641 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:22.642104 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641698 4991 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:22.642104 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641720 4991 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:22.642104 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641740 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641726458 +0000 UTC m=+4.941136321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:22.642104 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641755 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641748253 +0000 UTC m=+4.941158111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:22.642104 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641774 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:22.642104 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641794 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:22.642104 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:22.641860 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.641844289 +0000 UTC m=+4.941254162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:22.943478 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.943399 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5vnv5" event={"ID":"3ac87e61-9d62-4657-90a3-2bf7e4877f85","Type":"ContainerStarted","Data":"1612acee76e6eddc979fb13c09a46a09b25d0ffebe3829d6551df177c08d7e9e"} Oct 30 21:46:22.944648 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.944629 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbmj9" event={"ID":"dbdaddf0-875f-46f2-8418-62377b63a086","Type":"ContainerStarted","Data":"fc6a7afbd1015b47dce34ff2b0fa0a995d4e751aef0395151d2332e4e5cdea65"} Oct 30 21:46:22.946133 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.946112 4991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:22.946133 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:22.946121 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ssjmb" event={"ID":"74d26ea6-81b0-4ff7-acef-cd28751f4a78","Type":"ContainerStarted","Data":"a716c2376e5421a0623e92f510119efac29e66452d7a8950b49daa41b3da17b4"} Oct 30 21:46:23.950753 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:23.950688 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbmj9" event={"ID":"dbdaddf0-875f-46f2-8418-62377b63a086","Type":"ContainerStarted","Data":"93d760fb116edd99247bb9d5245968695c3b8a05cfb21c9fb707d3a61d128837"} Oct 30 21:46:23.950753 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:23.950740 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbmj9" event={"ID":"dbdaddf0-875f-46f2-8418-62377b63a086","Type":"ContainerStarted","Data":"0280429290ebb4cfe30479a42197243d29cb9c88b0548ae8e18b0a692629a55e"} Oct 30 21:46:24.660276 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660240 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:24.660276 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660280 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660305 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660346 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660371 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660392 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660408 4991 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660421 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.660466 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660467 4991 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660486 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls podName:6b4c949e-cff6-4abf-b3ff-f85046f76784 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:28.660467414 +0000 UTC m=+8.959877295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n456p" (UID: "6b4c949e-cff6-4abf-b3ff-f85046f76784") : secret "samples-operator-tls" not found Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660493 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660513 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs podName:a547a8cd-8909-475e-a03d-15b409877777 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:28.660503312 +0000 UTC m=+8.959913172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs") pod "router-default-5c8c9db44f-llgx4" (UID: "a547a8cd-8909-475e-a03d-15b409877777") : secret "router-metrics-certs-default" not found Oct 30 21:46:24.660522 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660513 4991 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660418 4991 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660615 4991 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660633 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d68ffbc4-r5mnd: secret "image-registry-tls" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660513 4991 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65fb6c59-872xp: secret "image-registry-tls" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660588 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert podName:4c494bc9-5908-48bc-9825-2669ad4196c1 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:28.660572749 +0000 UTC m=+8.959982619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j66r8" (UID: "4c494bc9-5908-48bc-9825-2669ad4196c1") : secret "networking-console-plugin-cert" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660720 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:28.660705881 +0000 UTC m=+8.960115752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660735 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls podName:39391909-c450-41dd-bf4a-51208f0e7c32 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:28.660726153 +0000 UTC m=+8.960136026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls") pod "image-registry-58d68ffbc4-r5mnd" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32") : secret "image-registry-tls" not found Oct 30 21:46:24.661036 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:24.660750 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls podName:0ebd5c04-08c6-45f9-a141-bbcfb08734cf nodeName:}" failed. No retries permitted until 2026-10-30 21:46:28.660743026 +0000 UTC m=+8.960152890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls") pod "image-registry-5b65fb6c59-872xp" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf") : secret "image-registry-tls" not found Oct 30 21:46:24.662852 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.662826 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8743ad27-5d90-4b7f-8012-6faf21548f97-metrics-tls\") pod \"dns-default-8fwrp\" (UID: \"8743ad27-5d90-4b7f-8012-6faf21548f97\") " pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:24.662943 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.662880 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67d09fe6-fb0a-43cc-8b20-98cdb22eea2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dvh4d\" (UID: \"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:24.815816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.815791 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" Oct 30 21:46:24.829344 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:24.829309 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:25.239252 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.239226 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d"] Oct 30 21:46:25.245881 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:25.245857 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d09fe6_fb0a_43cc_8b20_98cdb22eea2a.slice/crio-0e63b1f16e0c5a64656b01cea0c02882a472ff66ed335dc5b01c56a273a72a91 WatchSource:0}: Error finding container 0e63b1f16e0c5a64656b01cea0c02882a472ff66ed335dc5b01c56a273a72a91: Status 404 returned error can't find the container with id 0e63b1f16e0c5a64656b01cea0c02882a472ff66ed335dc5b01c56a273a72a91 Oct 30 21:46:25.247034 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.247014 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fwrp"] Oct 30 21:46:25.286059 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:25.286039 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8743ad27_5d90_4b7f_8012_6faf21548f97.slice/crio-7cacbf1f35116a23c1b15b2803198f3b07fbbfdf70cbd59946e0694a3d1f7498 WatchSource:0}: Error finding container 7cacbf1f35116a23c1b15b2803198f3b07fbbfdf70cbd59946e0694a3d1f7498: Status 404 returned error can't find the container with id 7cacbf1f35116a23c1b15b2803198f3b07fbbfdf70cbd59946e0694a3d1f7498 Oct 30 21:46:25.623813 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.623781 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:25.623984 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.623909 4991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:25.628737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.628715 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jqbm4" Oct 30 21:46:25.958312 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.958238 4991 generic.go:356] "Generic (PLEG): container finished" podID="d6594a6f-9c78-40bf-99b7-fb21cd031157" containerID="b59f77cc7ce0ff51bc5082a98e457a970cdef300c154434959dcd19a769ca6f5" exitCode=0 Oct 30 21:46:25.958482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.958335 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" event={"ID":"d6594a6f-9c78-40bf-99b7-fb21cd031157","Type":"ContainerDied","Data":"b59f77cc7ce0ff51bc5082a98e457a970cdef300c154434959dcd19a769ca6f5"} Oct 30 21:46:25.958701 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.958681 4991 scope.go:122] "RemoveContainer" containerID="b59f77cc7ce0ff51bc5082a98e457a970cdef300c154434959dcd19a769ca6f5" Oct 30 21:46:25.959625 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.959597 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" event={"ID":"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a","Type":"ContainerStarted","Data":"0e63b1f16e0c5a64656b01cea0c02882a472ff66ed335dc5b01c56a273a72a91"} Oct 30 21:46:25.960705 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:25.960686 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fwrp" event={"ID":"8743ad27-5d90-4b7f-8012-6faf21548f97","Type":"ContainerStarted","Data":"7cacbf1f35116a23c1b15b2803198f3b07fbbfdf70cbd59946e0694a3d1f7498"} Oct 30 21:46:26.969221 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:26.969187 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6kc9p" event={"ID":"d6594a6f-9c78-40bf-99b7-fb21cd031157","Type":"ContainerStarted","Data":"14ca1f2f894dc896a7825376b4708ce2f8829cdf1c3b26653e148688c25347a8"} Oct 30 21:46:27.439587 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:27.439554 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:27.974186 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:27.974154 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dvh4d" event={"ID":"67d09fe6-fb0a-43cc-8b20-98cdb22eea2a","Type":"ContainerStarted","Data":"9ca3a9addcf7756e16f90e2bb6f9a4561264f9eec1a233cf5aedaacd42fe0200"} Oct 30 21:46:27.976041 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:27.976020 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fwrp" event={"ID":"8743ad27-5d90-4b7f-8012-6faf21548f97","Type":"ContainerStarted","Data":"ec42e41e0f8ccb41d8fb1e1681157e25d51952ecc0cf362e43723019bdffacce"} Oct 30 21:46:27.976136 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:27.976059 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fwrp" event={"ID":"8743ad27-5d90-4b7f-8012-6faf21548f97","Type":"ContainerStarted","Data":"81f0df8cda095eff98ad236fdd2594ece0143056c1141eab63d50455f26a9717"} Oct 30 21:46:27.976136 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:27.976096 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:27.997079 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:27.997062 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r2dj5" Oct 30 21:46:28.121876 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.121853 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:28.122045 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.122033 4991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:28.697906 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.697876 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:28.697906 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.697911 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:28.698122 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.697934 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:28.698122 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.697959 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:28.698122 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.698007 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:28.698122 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:28.698049 4991 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:46:28.698122 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:28.698123 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls podName:7e021343-49b5-4868-a1f8-3a1bd283c42c nodeName:}" failed. No retries permitted until 2026-10-30 21:46:36.698104763 +0000 UTC m=+16.997514637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhs7l" (UID: "7e021343-49b5-4868-a1f8-3a1bd283c42c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:46:28.698405 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.698059 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:28.700737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.700686 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c494bc9-5908-48bc-9825-2669ad4196c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j66r8\" (UID: \"4c494bc9-5908-48bc-9825-2669ad4196c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:28.700874 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.700795 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"image-registry-58d68ffbc4-r5mnd\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:28.700874 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.700851 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a547a8cd-8909-475e-a03d-15b409877777-metrics-certs\") pod \"router-default-5c8c9db44f-llgx4\" (UID: \"a547a8cd-8909-475e-a03d-15b409877777\") " pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:28.700954 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.700899 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4c949e-cff6-4abf-b3ff-f85046f76784-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n456p\" (UID: \"6b4c949e-cff6-4abf-b3ff-f85046f76784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:28.701369 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.701350 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"image-registry-5b65fb6c59-872xp\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:28.712554 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.712534 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgx8f" Oct 30 21:46:28.716865 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.716837 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:28.716960 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.716928 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:28.717903 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.717888 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" Oct 30 21:46:28.719510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.719492 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" Oct 30 21:46:28.727642 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:28.727621 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:29.165053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.165018 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b65fb6c59-872xp"] Oct 30 21:46:29.171962 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:29.171939 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebd5c04_08c6_45f9_a141_bbcfb08734cf.slice/crio-833828f57255f98562af8402c327de8605cb4e08f67167f09e11bf5567d27142 WatchSource:0}: Error finding container 833828f57255f98562af8402c327de8605cb4e08f67167f09e11bf5567d27142: Status 404 returned error can't find the container with id 833828f57255f98562af8402c327de8605cb4e08f67167f09e11bf5567d27142 Oct 30 21:46:29.191380 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.191285 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d68ffbc4-r5mnd"] Oct 30 21:46:29.192053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.192034 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c8c9db44f-llgx4"] Oct 30 21:46:29.193076 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.193058 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p"] Oct 30 21:46:29.200102 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:29.200073 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39391909_c450_41dd_bf4a_51208f0e7c32.slice/crio-9ee7b6c193f5b66ab33b71bf78e06212222455c92405fbdbdad18c6b37ba814e WatchSource:0}: Error finding container 9ee7b6c193f5b66ab33b71bf78e06212222455c92405fbdbdad18c6b37ba814e: Status 404 returned error can't find the container with id 9ee7b6c193f5b66ab33b71bf78e06212222455c92405fbdbdad18c6b37ba814e Oct 30 21:46:29.200599 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:29.200532 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda547a8cd_8909_475e_a03d_15b409877777.slice/crio-75fa67eeb476d50b98f5422eefd87315bd4bfb3454651447367d1a1ed8940513 WatchSource:0}: Error finding container 75fa67eeb476d50b98f5422eefd87315bd4bfb3454651447367d1a1ed8940513: Status 404 returned error can't find the container with id 75fa67eeb476d50b98f5422eefd87315bd4bfb3454651447367d1a1ed8940513 Oct 30 21:46:29.214671 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.214643 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j66r8"] Oct 30 21:46:29.238554 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:29.238529 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c494bc9_5908_48bc_9825_2669ad4196c1.slice/crio-db5c8d02bc279024b7609d18670a218550f5ae8f46f8deeb44f22277075d0d03 WatchSource:0}: Error finding container db5c8d02bc279024b7609d18670a218550f5ae8f46f8deeb44f22277075d0d03: Status 404 returned error can't find the container with id db5c8d02bc279024b7609d18670a218550f5ae8f46f8deeb44f22277075d0d03 Oct 30 21:46:29.984251 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.984212 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" event={"ID":"0ebd5c04-08c6-45f9-a141-bbcfb08734cf","Type":"ContainerStarted","Data":"bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d"} Oct 30 21:46:29.984452 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.984272 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" event={"ID":"0ebd5c04-08c6-45f9-a141-bbcfb08734cf","Type":"ContainerStarted","Data":"833828f57255f98562af8402c327de8605cb4e08f67167f09e11bf5567d27142"} Oct 30 21:46:29.985792 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.985766 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" event={"ID":"39391909-c450-41dd-bf4a-51208f0e7c32","Type":"ContainerStarted","Data":"2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241"} Oct 30 21:46:29.985921 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.985809 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" event={"ID":"39391909-c450-41dd-bf4a-51208f0e7c32","Type":"ContainerStarted","Data":"9ee7b6c193f5b66ab33b71bf78e06212222455c92405fbdbdad18c6b37ba814e"} Oct 30 21:46:29.987029 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.987002 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" event={"ID":"4c494bc9-5908-48bc-9825-2669ad4196c1","Type":"ContainerStarted","Data":"db5c8d02bc279024b7609d18670a218550f5ae8f46f8deeb44f22277075d0d03"} Oct 30 21:46:29.988582 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.988550 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" event={"ID":"a547a8cd-8909-475e-a03d-15b409877777","Type":"ContainerStarted","Data":"4adbf8635f95b75358b0351d57b3424a72b91a3e8ee993b7efe6e7a475869606"} Oct 30 21:46:29.988665 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.988589 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" event={"ID":"a547a8cd-8909-475e-a03d-15b409877777","Type":"ContainerStarted","Data":"75fa67eeb476d50b98f5422eefd87315bd4bfb3454651447367d1a1ed8940513"} Oct 30 21:46:29.989665 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:29.989642 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" event={"ID":"6b4c949e-cff6-4abf-b3ff-f85046f76784","Type":"ContainerStarted","Data":"a05cef13d2f704275209859b78170f53edfaaf70c0de1e8692bba4b6729d1a44"} Oct 30 21:46:30.718153 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:30.718120 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:30.721031 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:30.721008 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:30.911302 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:30.911267 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:30.999082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:30.996688 4991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:30.999082 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:30.997287 4991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:32.001762 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:32.001728 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j66r8" event={"ID":"4c494bc9-5908-48bc-9825-2669ad4196c1","Type":"ContainerStarted","Data":"1f4a8e97b657a2f6f4c352056d649d5a126ce0c4dfb0951481ccde81e7b3c4df"} Oct 30 21:46:32.003756 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:32.003719 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" event={"ID":"6b4c949e-cff6-4abf-b3ff-f85046f76784","Type":"ContainerStarted","Data":"62977b549cb8076e0be3424372ff7646074b5ff2673222fdf31b297808e2d724"} Oct 30 21:46:32.003894 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:32.003780 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n456p" event={"ID":"6b4c949e-cff6-4abf-b3ff-f85046f76784","Type":"ContainerStarted","Data":"be46a3e6dd1e9e1d6d3cacce06dc399b16e10170bb3165b6b954a43d664fd301"} Oct 30 21:46:32.003894 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:32.003741 4991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:33.541304 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:33.541276 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:34.269366 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:34.269333 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:34.269554 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:34.269509 4991 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:34.270801 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:34.270777 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c8c9db44f-llgx4" Oct 30 21:46:36.205100 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:36.205070 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:36.759574 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:36.759540 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:36.761854 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:36.761827 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e021343-49b5-4868-a1f8-3a1bd283c42c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhs7l\" (UID: \"7e021343-49b5-4868-a1f8-3a1bd283c42c\") " pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:36.830562 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:36.830538 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhs7l" Oct 30 21:46:37.201737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:37.201568 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhs7l"] Oct 30 21:46:37.206221 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:37.206194 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e021343_49b5_4868_a1f8_3a1bd283c42c.slice/crio-165f56e2666a7af2c9d55fae4cfe2a26d6eec062b8663699d220ba586ec50c18 WatchSource:0}: Error finding container 165f56e2666a7af2c9d55fae4cfe2a26d6eec062b8663699d220ba586ec50c18: Status 404 returned error can't find the container with id 165f56e2666a7af2c9d55fae4cfe2a26d6eec062b8663699d220ba586ec50c18 Oct 30 21:46:37.980946 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:37.980918 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8fwrp" Oct 30 21:46:38.029898 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:38.029865 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhs7l" event={"ID":"7e021343-49b5-4868-a1f8-3a1bd283c42c","Type":"ContainerStarted","Data":"e3389b51f69edc3c111bc7c92922e49a5f2ae6c0f6e9081ecf8e1b1be1548c6a"} Oct 30 21:46:38.030025 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:38.029914 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhs7l" event={"ID":"7e021343-49b5-4868-a1f8-3a1bd283c42c","Type":"ContainerStarted","Data":"165f56e2666a7af2c9d55fae4cfe2a26d6eec062b8663699d220ba586ec50c18"} Oct 30 21:46:39.035776 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:39.035742 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhs7l" event={"ID":"7e021343-49b5-4868-a1f8-3a1bd283c42c","Type":"ContainerStarted","Data":"3febc5455a2f17025d9237cb98b7a571b6aaed5478d2c1a5f9441e7d36df6a36"} Oct 30 21:46:39.758855 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:39.758823 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b65fb6c59-872xp"] Oct 30 21:46:41.046065 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:41.046009 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhs7l" event={"ID":"7e021343-49b5-4868-a1f8-3a1bd283c42c","Type":"ContainerStarted","Data":"ed8829b466ed270f3a99efe081dd525b2c05c2abbb6c04ba5b10e2b5be68abe3"} Oct 30 21:46:46.209194 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:46.209160 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:46:47.909092 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:46:47.909058 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:46:49.764545 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:49.764518 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:46:50.913603 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:50.913576 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tqcc7" Oct 30 21:46:52.297846 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.297811 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2f47h"] Oct 30 21:46:52.328841 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.328822 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364607 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364580 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-wtmp\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364612 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-textfile\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364630 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364787 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364715 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-sys\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364787 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364763 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-tls\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364888 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364822 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9xpt\" (UniqueName: \"kubernetes.io/projected/1d16bd55-bd8f-42df-9bac-f93265002009-kube-api-access-s9xpt\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364935 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364917 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.364971 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.364953 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-root\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.365047 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.365033 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d16bd55-bd8f-42df-9bac-f93265002009-metrics-client-ca\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.406916 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.406895 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:46:52.431819 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.431774 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4gblc\"" Oct 30 21:46:52.431914 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.431820 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:46:52.431914 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.431900 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:46:52.431998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.431941 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:46:52.465868 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.465840 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d16bd55-bd8f-42df-9bac-f93265002009-metrics-client-ca\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.465973 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.465894 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-wtmp\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.465973 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.465919 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-textfile\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.465973 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.465946 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466131 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.465973 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-sys\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466131 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466000 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-tls\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466131 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466052 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-sys\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466131 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466054 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xpt\" (UniqueName: \"kubernetes.io/projected/1d16bd55-bd8f-42df-9bac-f93265002009-kube-api-access-s9xpt\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466182 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-wtmp\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466242 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466286 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-textfile\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466315 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-root\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466539 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466376 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d16bd55-bd8f-42df-9bac-f93265002009-root\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466539 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466512 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d16bd55-bd8f-42df-9bac-f93265002009-metrics-client-ca\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.466670 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.466649 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.468329 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.468290 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.468414 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.468337 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d16bd55-bd8f-42df-9bac-f93265002009-node-exporter-tls\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.526863 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.526839 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xpt\" (UniqueName: \"kubernetes.io/projected/1d16bd55-bd8f-42df-9bac-f93265002009-kube-api-access-s9xpt\") pod \"node-exporter-2f47h\" (UID: \"1d16bd55-bd8f-42df-9bac-f93265002009\") " pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.683247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:52.683174 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2f47h" Oct 30 21:46:52.706599 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:46:52.706576 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d16bd55_bd8f_42df_9bac_f93265002009.slice/crio-b99029abb0d52dddd00b44c2a8a60d0111b3e03a817b3a15062e924eafa64ab3 WatchSource:0}: Error finding container b99029abb0d52dddd00b44c2a8a60d0111b3e03a817b3a15062e924eafa64ab3: Status 404 returned error can't find the container with id b99029abb0d52dddd00b44c2a8a60d0111b3e03a817b3a15062e924eafa64ab3 Oct 30 21:46:53.091662 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:53.091628 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f47h" event={"ID":"1d16bd55-bd8f-42df-9bac-f93265002009","Type":"ContainerStarted","Data":"b99029abb0d52dddd00b44c2a8a60d0111b3e03a817b3a15062e924eafa64ab3"} Oct 30 21:46:55.102015 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:55.101981 4991 generic.go:356] "Generic (PLEG): container finished" podID="1d16bd55-bd8f-42df-9bac-f93265002009" containerID="0df533ec38f158cb788552444844e77cd2b6e1a4d04f459734fc2ff7fc01d3bb" exitCode=0 Oct 30 21:46:55.102416 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:55.102037 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f47h" event={"ID":"1d16bd55-bd8f-42df-9bac-f93265002009","Type":"ContainerDied","Data":"0df533ec38f158cb788552444844e77cd2b6e1a4d04f459734fc2ff7fc01d3bb"} Oct 30 21:46:56.108333 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:56.108295 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f47h" event={"ID":"1d16bd55-bd8f-42df-9bac-f93265002009","Type":"ContainerStarted","Data":"92c64609e98cc9439ae54881fe1ef4bead24741567fdcb42f7e924157d659928"} Oct 30 21:46:56.108782 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:56.108360 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f47h" event={"ID":"1d16bd55-bd8f-42df-9bac-f93265002009","Type":"ContainerStarted","Data":"5f948d3704ebfe67e159f7884a3275a62cabc1183b2342fd2cae406920075d98"} Oct 30 21:46:56.224589 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:46:56.224535 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2f47h" podStartSLOduration=2.585096097 podStartE2EDuration="4.224520998s" totalImagesPullingTime="1.571817781s" totalInitContainerRuntime="67.60712ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:56.223354894 +0000 UTC m=+36.522764774" watchObservedRunningTime="2026-10-30 21:46:56.224520998 +0000 UTC m=+36.523930875" Oct 30 21:47:04.262369 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:04.262282 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d68ffbc4-r5mnd"] Oct 30 21:47:04.783169 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:04.783130 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" podUID="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" containerName="registry" containerID="cri-o://bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d" gracePeriod=30 Oct 30 21:47:05.021552 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.021526 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:47:05.144571 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.144496 4991 generic.go:356] "Generic (PLEG): container finished" podID="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" containerID="bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d" exitCode=0 Oct 30 21:47:05.144571 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.144559 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" Oct 30 21:47:05.144748 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.144577 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" event={"ID":"0ebd5c04-08c6-45f9-a141-bbcfb08734cf","Type":"ContainerDied","Data":"bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d"} Oct 30 21:47:05.144748 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.144663 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65fb6c59-872xp" event={"ID":"0ebd5c04-08c6-45f9-a141-bbcfb08734cf","Type":"ContainerDied","Data":"833828f57255f98562af8402c327de8605cb4e08f67167f09e11bf5567d27142"} Oct 30 21:47:05.144748 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.144686 4991 scope.go:122] "RemoveContainer" containerID="bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d" Oct 30 21:47:05.152390 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.152374 4991 scope.go:122] "RemoveContainer" containerID="bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d" Oct 30 21:47:05.152665 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:47:05.152646 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d\": container with ID starting with bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d not found: ID does not exist" containerID="bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d" Oct 30 21:47:05.152714 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.152673 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d"} err="failed to get container status \"bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d\": rpc error: code = NotFound desc = could not find container \"bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d\": container with ID starting with bcf9735b13e84afcefbd595bf89203a0378b31a511fd20d2edd9af6c194b1f8d not found: ID does not exist" Oct 30 21:47:05.165940 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.165915 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.165945 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.165973 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.166018 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166160 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.166050 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166160 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.166089 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166160 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.166142 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166312 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.166191 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\") pod \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\" (UID: \"0ebd5c04-08c6-45f9-a141-bbcfb08734cf\") " Oct 30 21:47:05.166966 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.166554 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:05.166966 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.166895 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:05.168328 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.168267 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:05.168502 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.168483 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:05.168793 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.168762 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:05.168858 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.168826 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "kube-api-access-jhqdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:05.168858 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.168837 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:05.173583 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.173540 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted" pod "0ebd5c04-08c6-45f9-a141-bbcfb08734cf" (UID: "0ebd5c04-08c6-45f9-a141-bbcfb08734cf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.266928 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.266891 4991 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-bound-sa-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.266928 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.266928 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqdn\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-kube-api-access-jhqdn\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.267280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.266944 4991 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-trusted-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.267280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.266959 4991 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-certificates\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.267280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.266974 4991 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-ca-trust-extracted\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.267280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.266988 4991 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-image-registry-private-configuration\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.267280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.267003 4991 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-installation-pull-secrets\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.267280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.267017 4991 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebd5c04-08c6-45f9-a141-bbcfb08734cf-registry-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:05.558587 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.558555 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b65fb6c59-872xp"] Oct 30 21:47:05.605057 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.605031 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b65fb6c59-872xp"] Oct 30 21:47:05.873655 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:05.873618 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" path="/var/lib/kubelet/pods/0ebd5c04-08c6-45f9-a141-bbcfb08734cf/volumes" Oct 30 21:47:17.908789 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:47:17.908751 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:47:25.227475 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:25.227444 4991 generic.go:356] "Generic (PLEG): container finished" podID="4f3afb35-bfde-45ec-a972-78a6e7461d9d" containerID="0318d79587cf816f7900abf9e720c0e26aa372c17706afec28febb209bb4111d" exitCode=0 Oct 30 21:47:25.227804 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:25.227498 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" event={"ID":"4f3afb35-bfde-45ec-a972-78a6e7461d9d","Type":"ContainerDied","Data":"0318d79587cf816f7900abf9e720c0e26aa372c17706afec28febb209bb4111d"} Oct 30 21:47:25.227804 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:25.227743 4991 scope.go:122] "RemoveContainer" containerID="0318d79587cf816f7900abf9e720c0e26aa372c17706afec28febb209bb4111d" Oct 30 21:47:26.233429 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:26.233392 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zqp9c" event={"ID":"4f3afb35-bfde-45ec-a972-78a6e7461d9d","Type":"ContainerStarted","Data":"edf5fe40d71caa971355deb50be53ee2c6665272bf61c784bcdb06a11a84f598"} Oct 30 21:47:29.283471 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.283437 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" podUID="39391909-c450-41dd-bf4a-51208f0e7c32" containerName="registry" containerID="cri-o://2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241" gracePeriod=30 Oct 30 21:47:29.537521 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.537469 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:47:29.634763 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.634737 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.634886 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.634803 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.634886 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.634824 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.634886 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.634850 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.635043 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.635014 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.635163 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.635074 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.635163 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.635113 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.635277 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.635170 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\") pod \"39391909-c450-41dd-bf4a-51208f0e7c32\" (UID: \"39391909-c450-41dd-bf4a-51208f0e7c32\") " Oct 30 21:47:29.635277 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.635231 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:29.635610 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.635553 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:29.635610 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.635578 4991 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-registry-certificates\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.637232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.637202 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:29.637431 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.637379 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:29.637572 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.637549 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:29.637678 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.637656 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "kube-api-access-sgvtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:29.637720 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.637695 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:29.642046 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.642020 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted" pod "39391909-c450-41dd-bf4a-51208f0e7c32" (UID: "39391909-c450-41dd-bf4a-51208f0e7c32"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:29.735879 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.735852 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvtf\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-kube-api-access-sgvtf\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.735879 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.735875 4991 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-registry-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.736027 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.735886 4991 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39391909-c450-41dd-bf4a-51208f0e7c32-ca-trust-extracted\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.736027 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.735895 4991 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-installation-pull-secrets\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.736027 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.735907 4991 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39391909-c450-41dd-bf4a-51208f0e7c32-image-registry-private-configuration\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.736027 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.735917 4991 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39391909-c450-41dd-bf4a-51208f0e7c32-trusted-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.736027 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:29.735925 4991 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39391909-c450-41dd-bf4a-51208f0e7c32-bound-sa-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:30.252630 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.252597 4991 generic.go:356] "Generic (PLEG): container finished" podID="39391909-c450-41dd-bf4a-51208f0e7c32" containerID="2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241" exitCode=0 Oct 30 21:47:30.252630 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.252631 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" event={"ID":"39391909-c450-41dd-bf4a-51208f0e7c32","Type":"ContainerDied","Data":"2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241"} Oct 30 21:47:30.252842 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.252642 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" Oct 30 21:47:30.252842 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.252664 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d68ffbc4-r5mnd" event={"ID":"39391909-c450-41dd-bf4a-51208f0e7c32","Type":"ContainerDied","Data":"9ee7b6c193f5b66ab33b71bf78e06212222455c92405fbdbdad18c6b37ba814e"} Oct 30 21:47:30.252842 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.252686 4991 scope.go:122] "RemoveContainer" containerID="2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241" Oct 30 21:47:30.260565 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.260546 4991 scope.go:122] "RemoveContainer" containerID="2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241" Oct 30 21:47:30.260828 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:47:30.260810 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241\": container with ID starting with 2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241 not found: ID does not exist" containerID="2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241" Oct 30 21:47:30.260885 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.260840 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241"} err="failed to get container status \"2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241\": rpc error: code = NotFound desc = could not find container \"2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241\": container with ID starting with 2421054e4535d13c51872ee168f2b166b32eb543fb3e9f4c737349d35234d241 not found: ID does not exist" Oct 30 21:47:30.454633 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.454604 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d68ffbc4-r5mnd"] Oct 30 21:47:30.501633 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:30.501608 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58d68ffbc4-r5mnd"] Oct 30 21:47:31.872868 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:31.872833 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39391909-c450-41dd-bf4a-51208f0e7c32" path="/var/lib/kubelet/pods/39391909-c450-41dd-bf4a-51208f0e7c32/volumes" Oct 30 21:47:39.289107 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:39.289073 4991 generic.go:356] "Generic (PLEG): container finished" podID="a07200b0-a7f6-4c5a-9197-b73779a06c1f" containerID="2cfc479fb7215b646116e099071808a6ca897c5b3fa4c932c37b65f16df57f28" exitCode=0 Oct 30 21:47:39.289521 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:39.289118 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc7w4" event={"ID":"a07200b0-a7f6-4c5a-9197-b73779a06c1f","Type":"ContainerDied","Data":"2cfc479fb7215b646116e099071808a6ca897c5b3fa4c932c37b65f16df57f28"} Oct 30 21:47:39.289521 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:39.289454 4991 scope.go:122] "RemoveContainer" containerID="2cfc479fb7215b646116e099071808a6ca897c5b3fa4c932c37b65f16df57f28" Oct 30 21:47:40.254982 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.254945 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-p8p6k-4z6rv"] Oct 30 21:47:40.255182 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.255119 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" podUID="fbd83615-d417-4d88-88d2-9f74ffbce102" containerName="insights-gathering" containerID="cri-o://553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510" gracePeriod=30 Oct 30 21:47:40.280882 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:47:40.280855 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd83615_d417_4d88_88d2_9f74ffbce102.slice/crio-553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd83615_d417_4d88_88d2_9f74ffbce102.slice/crio-conmon-553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:47:40.294855 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.294830 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc7w4" event={"ID":"a07200b0-a7f6-4c5a-9197-b73779a06c1f","Type":"ContainerStarted","Data":"82a9029723a58e90d73f86a1b10f8b8053d71e97e518ca8693edbbb9e39443cb"} Oct 30 21:47:40.506734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.506667 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:47:40.611305 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.611272 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\") pod \"fbd83615-d417-4d88-88d2-9f74ffbce102\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " Oct 30 21:47:40.611466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.611338 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\") pod \"fbd83615-d417-4d88-88d2-9f74ffbce102\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " Oct 30 21:47:40.611466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.611360 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\" (UniqueName: \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\") pod \"fbd83615-d417-4d88-88d2-9f74ffbce102\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " Oct 30 21:47:40.611466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.611434 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\") pod \"fbd83615-d417-4d88-88d2-9f74ffbce102\" (UID: \"fbd83615-d417-4d88-88d2-9f74ffbce102\") " Oct 30 21:47:40.611688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.611667 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path" pod "fbd83615-d417-4d88-88d2-9f74ffbce102" (UID: "fbd83615-d417-4d88-88d2-9f74ffbce102"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:40.611726 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.611650 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle" pod "fbd83615-d417-4d88-88d2-9f74ffbce102" (UID: "fbd83615-d417-4d88-88d2-9f74ffbce102"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:40.611816 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.611798 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle" pod "fbd83615-d417-4d88-88d2-9f74ffbce102" (UID: "fbd83615-d417-4d88-88d2-9f74ffbce102"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:40.613626 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.613598 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8" pod "fbd83615-d417-4d88-88d2-9f74ffbce102" (UID: "fbd83615-d417-4d88-88d2-9f74ffbce102"). InnerVolumeSpecName "kube-api-access-jcdh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:40.663046 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663020 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-75zlj-727nt"] Oct 30 21:47:40.663313 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663302 4991 cpu_manager.go:414] "containerMap: removing container" podUID="fbd83615-d417-4d88-88d2-9f74ffbce102" containerName="insights-gathering" Oct 30 21:47:40.663374 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663315 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd83615-d417-4d88-88d2-9f74ffbce102" containerName="insights-gathering" Oct 30 21:47:40.663374 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663364 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" containerName="registry" Oct 30 21:47:40.663374 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663370 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" containerName="registry" Oct 30 21:47:40.663482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663381 4991 cpu_manager.go:414] "containerMap: removing container" podUID="39391909-c450-41dd-bf4a-51208f0e7c32" containerName="registry" Oct 30 21:47:40.663482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663386 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39391909-c450-41dd-bf4a-51208f0e7c32" containerName="registry" Oct 30 21:47:40.663482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663440 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebd5c04-08c6-45f9-a141-bbcfb08734cf" containerName="registry" Oct 30 21:47:40.663482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663449 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="39391909-c450-41dd-bf4a-51208f0e7c32" containerName="registry" Oct 30 21:47:40.663482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.663456 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd83615-d417-4d88-88d2-9f74ffbce102" containerName="insights-gathering" Oct 30 21:47:40.666510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.666484 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.712638 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.712611 4991 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-service-ca-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:40.712638 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.712635 4991 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fbd83615-d417-4d88-88d2-9f74ffbce102-archives-path\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:40.712774 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.712646 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdh8\" (UniqueName: \"kubernetes.io/projected/fbd83615-d417-4d88-88d2-9f74ffbce102-kube-api-access-jcdh8\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:40.712774 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.712657 4991 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbd83615-d417-4d88-88d2-9f74ffbce102-trusted-ca-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:40.728879 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.728854 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-75zlj-727nt"] Oct 30 21:47:40.813458 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.813375 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-trusted-ca-bundle\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.813458 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.813433 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/36790a49-4cc4-4503-a33e-f54b5b86fa8e-archives-path\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.813660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.813522 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkqw\" (UniqueName: \"kubernetes.io/projected/36790a49-4cc4-4503-a33e-f54b5b86fa8e-kube-api-access-hdkqw\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.813660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.813574 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-service-ca-bundle\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.914296 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.914262 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-trusted-ca-bundle\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.914500 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.914314 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/36790a49-4cc4-4503-a33e-f54b5b86fa8e-archives-path\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.914500 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.914359 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkqw\" (UniqueName: \"kubernetes.io/projected/36790a49-4cc4-4503-a33e-f54b5b86fa8e-kube-api-access-hdkqw\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.914500 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.914386 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-service-ca-bundle\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.914784 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.914764 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/36790a49-4cc4-4503-a33e-f54b5b86fa8e-archives-path\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.915053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.915035 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-trusted-ca-bundle\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.915090 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.915036 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-service-ca-bundle\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.973720 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.973693 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkqw\" (UniqueName: \"kubernetes.io/projected/36790a49-4cc4-4503-a33e-f54b5b86fa8e-kube-api-access-hdkqw\") pod \"periodic-gathering-75zlj-727nt\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:40.977479 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:40.977456 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:47:41.300212 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.300182 4991 generic.go:356] "Generic (PLEG): container finished" podID="fbd83615-d417-4d88-88d2-9f74ffbce102" containerID="553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510" exitCode=2 Oct 30 21:47:41.300602 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.300227 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" event={"ID":"fbd83615-d417-4d88-88d2-9f74ffbce102","Type":"ContainerDied","Data":"553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510"} Oct 30 21:47:41.300602 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.300263 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" event={"ID":"fbd83615-d417-4d88-88d2-9f74ffbce102","Type":"ContainerDied","Data":"5b2834773e61d387dc57513e5a041518290fbbedb9765b15e5eb798106f953a4"} Oct 30 21:47:41.300602 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.300282 4991 scope.go:122] "RemoveContainer" containerID="553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510" Oct 30 21:47:41.300602 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.300228 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p8p6k-4z6rv" Oct 30 21:47:41.313593 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.313576 4991 scope.go:122] "RemoveContainer" containerID="553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510" Oct 30 21:47:41.313847 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:47:41.313828 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510\": container with ID starting with 553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510 not found: ID does not exist" containerID="553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510" Oct 30 21:47:41.313896 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.313855 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510"} err="failed to get container status \"553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510\": rpc error: code = NotFound desc = could not find container \"553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510\": container with ID starting with 553ea6848c02e6d3a67079f2f65fbb7bb6c3f4258ece494c7851c037af670510 not found: ID does not exist" Oct 30 21:47:41.343172 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.343150 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-75zlj-727nt"] Oct 30 21:47:41.349755 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:47:41.349734 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36790a49_4cc4_4503_a33e_f54b5b86fa8e.slice/crio-15cc05c5e12a51dc144af5952f950ec8803f20498343a38e25676a815a161bd2 WatchSource:0}: Error finding container 15cc05c5e12a51dc144af5952f950ec8803f20498343a38e25676a815a161bd2: Status 404 returned error can't find the container with id 15cc05c5e12a51dc144af5952f950ec8803f20498343a38e25676a815a161bd2 Oct 30 21:47:41.469936 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.469903 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-p8p6k-4z6rv"] Oct 30 21:47:41.471998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.471978 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-p8p6k-4z6rv"] Oct 30 21:47:41.871762 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:41.871730 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd83615-d417-4d88-88d2-9f74ffbce102" path="/var/lib/kubelet/pods/fbd83615-d417-4d88-88d2-9f74ffbce102/volumes" Oct 30 21:47:42.307050 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:42.307016 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-75zlj-727nt" event={"ID":"36790a49-4cc4-4503-a33e-f54b5b86fa8e","Type":"ContainerStarted","Data":"215b3c1f2a8d6756f1110be5afb110372b175d445d6401599586e23b38022474"} Oct 30 21:47:42.307438 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:42.307060 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-75zlj-727nt" event={"ID":"36790a49-4cc4-4503-a33e-f54b5b86fa8e","Type":"ContainerStarted","Data":"15cc05c5e12a51dc144af5952f950ec8803f20498343a38e25676a815a161bd2"} Oct 30 21:47:42.416040 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:47:42.415988 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-75zlj-727nt" podStartSLOduration=2.415975008 podStartE2EDuration="2.415975008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:42.415586511 +0000 UTC m=+82.714996388" watchObservedRunningTime="2026-10-30 21:47:42.415975008 +0000 UTC m=+82.715384887" Oct 30 21:47:47.909950 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:47:47.909911 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:48:17.909088 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:48:17.909048 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:48:47.909834 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:48:47.909797 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:49:07.629652 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:07.629617 4991 generic.go:356] "Generic (PLEG): container finished" podID="36790a49-4cc4-4503-a33e-f54b5b86fa8e" containerID="215b3c1f2a8d6756f1110be5afb110372b175d445d6401599586e23b38022474" exitCode=0 Oct 30 21:49:07.630008 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:07.629689 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-75zlj-727nt" event={"ID":"36790a49-4cc4-4503-a33e-f54b5b86fa8e","Type":"ContainerDied","Data":"215b3c1f2a8d6756f1110be5afb110372b175d445d6401599586e23b38022474"} Oct 30 21:49:08.759731 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.759710 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:49:08.837442 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.837415 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36790a49-4cc4-4503-a33e-f54b5b86fa8e-kube-api-access-hdkqw\" (UniqueName: \"kubernetes.io/projected/36790a49-4cc4-4503-a33e-f54b5b86fa8e-kube-api-access-hdkqw\") pod \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " Oct 30 21:49:08.837573 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.837456 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-trusted-ca-bundle\") pod \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " Oct 30 21:49:08.837573 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.837505 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36790a49-4cc4-4503-a33e-f54b5b86fa8e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/36790a49-4cc4-4503-a33e-f54b5b86fa8e-archives-path\") pod \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " Oct 30 21:49:08.837686 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.837587 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-service-ca-bundle\") pod \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\" (UID: \"36790a49-4cc4-4503-a33e-f54b5b86fa8e\") " Oct 30 21:49:08.837897 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.837869 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36790a49-4cc4-4503-a33e-f54b5b86fa8e-archives-path" pod "36790a49-4cc4-4503-a33e-f54b5b86fa8e" (UID: "36790a49-4cc4-4503-a33e-f54b5b86fa8e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:08.837978 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.837876 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-trusted-ca-bundle" pod "36790a49-4cc4-4503-a33e-f54b5b86fa8e" (UID: "36790a49-4cc4-4503-a33e-f54b5b86fa8e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:08.837978 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.837935 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-service-ca-bundle" pod "36790a49-4cc4-4503-a33e-f54b5b86fa8e" (UID: "36790a49-4cc4-4503-a33e-f54b5b86fa8e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:08.839424 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.839400 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36790a49-4cc4-4503-a33e-f54b5b86fa8e-kube-api-access-hdkqw" pod "36790a49-4cc4-4503-a33e-f54b5b86fa8e" (UID: "36790a49-4cc4-4503-a33e-f54b5b86fa8e"). InnerVolumeSpecName "kube-api-access-hdkqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:08.938094 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.938010 4991 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-service-ca-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:08.938094 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.938066 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkqw\" (UniqueName: \"kubernetes.io/projected/36790a49-4cc4-4503-a33e-f54b5b86fa8e-kube-api-access-hdkqw\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:08.938094 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.938077 4991 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36790a49-4cc4-4503-a33e-f54b5b86fa8e-trusted-ca-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:08.938094 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:08.938087 4991 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/36790a49-4cc4-4503-a33e-f54b5b86fa8e-archives-path\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:09.639412 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:09.639374 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-75zlj-727nt" event={"ID":"36790a49-4cc4-4503-a33e-f54b5b86fa8e","Type":"ContainerDied","Data":"15cc05c5e12a51dc144af5952f950ec8803f20498343a38e25676a815a161bd2"} Oct 30 21:49:09.639586 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:09.639453 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15cc05c5e12a51dc144af5952f950ec8803f20498343a38e25676a815a161bd2" Oct 30 21:49:09.639586 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:49:09.639393 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-75zlj-727nt" Oct 30 21:49:17.909592 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:49:17.909555 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:49:47.909999 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:49:47.909966 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:50:17.909338 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:50:17.909269 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:50:47.909444 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:50:47.909407 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:51:01.960110 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:01.960078 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw"] Oct 30 21:51:01.960577 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:01.960388 4991 cpu_manager.go:414] "containerMap: removing container" podUID="36790a49-4cc4-4503-a33e-f54b5b86fa8e" containerName="insights-gathering" Oct 30 21:51:01.960577 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:01.960399 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36790a49-4cc4-4503-a33e-f54b5b86fa8e" containerName="insights-gathering" Oct 30 21:51:01.960577 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:01.960464 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="36790a49-4cc4-4503-a33e-f54b5b86fa8e" containerName="insights-gathering" Oct 30 21:51:01.963519 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:01.963500 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.000727 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.000696 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgkzw\" (UniqueName: \"kubernetes.io/projected/f0d3536e-9ac4-48e1-8647-3370c950e0a3-kube-api-access-pgkzw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.000859 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.000738 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.000859 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.000755 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.012769 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.012749 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:51:02.059495 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.059463 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:51:02.059645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.059561 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-57gjd\"" Oct 30 21:51:02.060622 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.060605 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw"] Oct 30 21:51:02.101617 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.101592 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.101742 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.101620 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.101742 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.101676 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkzw\" (UniqueName: \"kubernetes.io/projected/f0d3536e-9ac4-48e1-8647-3370c950e0a3-kube-api-access-pgkzw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.101969 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.101947 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.102041 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.102005 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.159718 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.159690 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkzw\" (UniqueName: \"kubernetes.io/projected/f0d3536e-9ac4-48e1-8647-3370c950e0a3-kube-api-access-pgkzw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.278623 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.278594 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:02.646364 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:02.646264 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw"] Oct 30 21:51:02.653453 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:51:02.653420 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d3536e_9ac4_48e1_8647_3370c950e0a3.slice/crio-7f9864d02152b68a43197ce230fc61c7c830fb83ac507ca4ff5205562baeca1c WatchSource:0}: Error finding container 7f9864d02152b68a43197ce230fc61c7c830fb83ac507ca4ff5205562baeca1c: Status 404 returned error can't find the container with id 7f9864d02152b68a43197ce230fc61c7c830fb83ac507ca4ff5205562baeca1c Oct 30 21:51:03.062751 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.062713 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" event={"ID":"f0d3536e-9ac4-48e1-8647-3370c950e0a3","Type":"ContainerStarted","Data":"7f9864d02152b68a43197ce230fc61c7c830fb83ac507ca4ff5205562baeca1c"} Oct 30 21:51:03.445167 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.445077 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q"] Oct 30 21:51:03.449668 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.449644 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.512026 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.511996 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.512182 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.512036 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.512182 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.512116 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxx9v\" (UniqueName: \"kubernetes.io/projected/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-kube-api-access-fxx9v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.545479 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.545442 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-qsdjl\"" Oct 30 21:51:03.546712 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.546688 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q"] Oct 30 21:51:03.547549 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.547528 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz"] Oct 30 21:51:03.550877 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.550862 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.600160 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.600127 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-wbxpj\"" Oct 30 21:51:03.605175 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.605147 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz"] Oct 30 21:51:03.613113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613079 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbzc\" (UniqueName: \"kubernetes.io/projected/2130981f-7ad9-49c5-9f44-b5d97c822b0c-kube-api-access-qlbzc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.613245 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613210 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.613310 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613249 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.613310 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613289 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxx9v\" (UniqueName: \"kubernetes.io/projected/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-kube-api-access-fxx9v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.613433 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613353 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.613433 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613419 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.613627 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613607 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.613698 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.613678 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.650212 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.650185 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq"] Oct 30 21:51:03.653762 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.653744 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.670940 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.670913 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxx9v\" (UniqueName: \"kubernetes.io/projected/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-kube-api-access-fxx9v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.708126 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.708060 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-tp7nt\"" Oct 30 21:51:03.709379 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.709349 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq"] Oct 30 21:51:03.714041 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714017 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.714142 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714047 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.714142 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714074 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.714142 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714099 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbzc\" (UniqueName: \"kubernetes.io/projected/2130981f-7ad9-49c5-9f44-b5d97c822b0c-kube-api-access-qlbzc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.714333 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714199 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhxk\" (UniqueName: \"kubernetes.io/projected/38ba858f-337c-4848-ae6f-80bbdaa19b12-kube-api-access-dvhxk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.714333 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714240 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.714443 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714423 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.714479 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.714453 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.745492 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.745468 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj"] Oct 30 21:51:03.748972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.748952 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.766008 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.765984 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:03.772275 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.772246 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbzc\" (UniqueName: \"kubernetes.io/projected/2130981f-7ad9-49c5-9f44-b5d97c822b0c-kube-api-access-qlbzc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.804614 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.804581 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj"] Oct 30 21:51:03.808910 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.808881 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lvp5b\"" Oct 30 21:51:03.815059 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815031 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.815262 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815081 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.815262 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815117 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnz6\" (UniqueName: \"kubernetes.io/projected/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-kube-api-access-vnnz6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.815262 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815153 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.815262 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815209 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhxk\" (UniqueName: \"kubernetes.io/projected/38ba858f-337c-4848-ae6f-80bbdaa19b12-kube-api-access-dvhxk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.815262 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815237 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.815615 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815536 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.815681 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.815632 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.866402 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.866374 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:03.878399 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.877102 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhxk\" (UniqueName: \"kubernetes.io/projected/38ba858f-337c-4848-ae6f-80bbdaa19b12-kube-api-access-dvhxk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.917358 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.916709 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.917358 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.916773 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnz6\" (UniqueName: \"kubernetes.io/projected/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-kube-api-access-vnnz6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.917358 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.916802 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.917358 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.917148 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.917358 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.917261 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:03.970566 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.970499 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:03.975674 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:03.975649 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnz6\" (UniqueName: \"kubernetes.io/projected/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-kube-api-access-vnnz6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:04.067063 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:04.067032 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:04.146035 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:04.146005 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q"] Oct 30 21:51:04.155313 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:51:04.155285 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e030d2_d8cd_4dd2_87a2_1c47c4d414bc.slice/crio-e8346058ac3d1840fb77c3a2f590141a727930630da2c45c656d788a4fa5299a WatchSource:0}: Error finding container e8346058ac3d1840fb77c3a2f590141a727930630da2c45c656d788a4fa5299a: Status 404 returned error can't find the container with id e8346058ac3d1840fb77c3a2f590141a727930630da2c45c656d788a4fa5299a Oct 30 21:51:04.248632 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:04.248602 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz"] Oct 30 21:51:04.255739 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:51:04.255714 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2130981f_7ad9_49c5_9f44_b5d97c822b0c.slice/crio-a5ecf93df8f83561f770925f6f1404668c9f2db91cb1cdc2d26a44ed2ae90e79 WatchSource:0}: Error finding container a5ecf93df8f83561f770925f6f1404668c9f2db91cb1cdc2d26a44ed2ae90e79: Status 404 returned error can't find the container with id a5ecf93df8f83561f770925f6f1404668c9f2db91cb1cdc2d26a44ed2ae90e79 Oct 30 21:51:04.344984 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:04.344952 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq"] Oct 30 21:51:04.352091 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:51:04.352060 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ba858f_337c_4848_ae6f_80bbdaa19b12.slice/crio-59ff9a66ee6b4aa971923f3b547e16240359bf81efe95f775c566b5e3a4e7072 WatchSource:0}: Error finding container 59ff9a66ee6b4aa971923f3b547e16240359bf81efe95f775c566b5e3a4e7072: Status 404 returned error can't find the container with id 59ff9a66ee6b4aa971923f3b547e16240359bf81efe95f775c566b5e3a4e7072 Oct 30 21:51:04.442626 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:04.442599 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj"] Oct 30 21:51:04.448712 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:51:04.448683 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03f3aab_40d5_4ba7_b45a_2e7a334b0353.slice/crio-f93b0a2ef203a3a7fa3f01b914e742c8c75034b1c76bffaac0d184887e26c053 WatchSource:0}: Error finding container f93b0a2ef203a3a7fa3f01b914e742c8c75034b1c76bffaac0d184887e26c053: Status 404 returned error can't find the container with id f93b0a2ef203a3a7fa3f01b914e742c8c75034b1c76bffaac0d184887e26c053 Oct 30 21:51:05.074596 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.074559 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" event={"ID":"38ba858f-337c-4848-ae6f-80bbdaa19b12","Type":"ContainerStarted","Data":"59ff9a66ee6b4aa971923f3b547e16240359bf81efe95f775c566b5e3a4e7072"} Oct 30 21:51:05.075954 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.075921 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" event={"ID":"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc","Type":"ContainerStarted","Data":"e8346058ac3d1840fb77c3a2f590141a727930630da2c45c656d788a4fa5299a"} Oct 30 21:51:05.077287 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.077236 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" event={"ID":"d03f3aab-40d5-4ba7-b45a-2e7a334b0353","Type":"ContainerStarted","Data":"f93b0a2ef203a3a7fa3f01b914e742c8c75034b1c76bffaac0d184887e26c053"} Oct 30 21:51:05.078475 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.078449 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" event={"ID":"2130981f-7ad9-49c5-9f44-b5d97c822b0c","Type":"ContainerStarted","Data":"a5ecf93df8f83561f770925f6f1404668c9f2db91cb1cdc2d26a44ed2ae90e79"} Oct 30 21:51:05.175261 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.175234 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8"] Oct 30 21:51:05.179022 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.179000 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.227481 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.227456 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.227612 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.227524 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2wg\" (UniqueName: \"kubernetes.io/projected/c3282554-842e-4103-a48c-b1c201a7ca9d-kube-api-access-wh2wg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.227665 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.227623 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.228821 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.228693 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ctvtr\"" Oct 30 21:51:05.276724 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.276696 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8"] Oct 30 21:51:05.328406 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.328342 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.328527 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.328418 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2wg\" (UniqueName: \"kubernetes.io/projected/c3282554-842e-4103-a48c-b1c201a7ca9d-kube-api-access-wh2wg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.328527 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.328459 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.328864 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.328841 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.329099 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.329082 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.395360 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.388609 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2wg\" (UniqueName: \"kubernetes.io/projected/c3282554-842e-4103-a48c-b1c201a7ca9d-kube-api-access-wh2wg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.498085 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.498051 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:05.519666 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.519634 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx"] Oct 30 21:51:05.527598 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.527572 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.580234 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.578221 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7xsr9\"" Oct 30 21:51:05.580234 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.580128 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx"] Oct 30 21:51:05.632097 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.632069 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.632232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.632127 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.632232 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.632172 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnhd\" (UniqueName: \"kubernetes.io/projected/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-kube-api-access-vcnhd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.734066 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.733574 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.734066 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.733662 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.734066 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.733703 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnhd\" (UniqueName: \"kubernetes.io/projected/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-kube-api-access-vcnhd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.734066 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.733966 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.734439 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.734155 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.792291 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.792260 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnhd\" (UniqueName: \"kubernetes.io/projected/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-kube-api-access-vcnhd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.850881 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.850812 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:05.924354 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:05.924299 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8"] Oct 30 21:51:05.932736 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:51:05.932690 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3282554_842e_4103_a48c_b1c201a7ca9d.slice/crio-bd3939c88db5290f598cbb04f551527a179eefb3151a559fd68d3fd0b453f9f7 WatchSource:0}: Error finding container bd3939c88db5290f598cbb04f551527a179eefb3151a559fd68d3fd0b453f9f7: Status 404 returned error can't find the container with id bd3939c88db5290f598cbb04f551527a179eefb3151a559fd68d3fd0b453f9f7 Oct 30 21:51:06.086033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:06.085996 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" event={"ID":"c3282554-842e-4103-a48c-b1c201a7ca9d","Type":"ContainerStarted","Data":"bd3939c88db5290f598cbb04f551527a179eefb3151a559fd68d3fd0b453f9f7"} Oct 30 21:51:06.322716 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:06.322675 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx"] Oct 30 21:51:07.096746 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.096703 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" event={"ID":"75a7f143-c4f5-42e6-8275-5c3d12c17b1e","Type":"ContainerStarted","Data":"f77b8a4f1b7c2ff98eed77b9b89c36fc0509e9adb6afa738bb695f8837d73c4c"} Oct 30 21:51:07.890305 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.890272 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z"] Oct 30 21:51:07.894521 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.894114 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.945126 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.945061 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z"] Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953374 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953423 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953441 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953478 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953495 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2wv\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-kube-api-access-xz2wv\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953520 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953548 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953577 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.953734 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.953593 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:07.959050 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.958932 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:51:07.988618 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:07.988564 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-twq7g\"" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057828 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057857 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057882 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057912 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057930 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057965 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057981 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2wv\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-kube-api-access-xz2wv\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.057998 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.058544 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.058025 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.061234 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.060740 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.061234 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.060930 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.061234 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.061068 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.061234 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.061196 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.061655 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.061604 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.074717 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.066563 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.082014 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.081957 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.120051 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.120024 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.122144 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.122115 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2wv\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-kube-api-access-xz2wv\") pod \"maas-default-gateway-openshift-default-6f54779779-fj82z\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.176587 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.174850 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph"] Oct 30 21:51:08.179969 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.179945 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.218449 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.218180 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:08.229753 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.229730 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-jpqxw\"" Oct 30 21:51:08.244003 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.242588 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph"] Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259428 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259474 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259506 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259531 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259597 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259627 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hmt\" (UniqueName: \"kubernetes.io/projected/ec12978d-ab70-452f-8b7b-4954fa7fce4c-kube-api-access-g4hmt\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259666 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259690 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.260007 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.259727 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.360954 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.360915 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361083 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.360975 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361083 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361002 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361083 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361028 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361096 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361125 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hmt\" (UniqueName: \"kubernetes.io/projected/ec12978d-ab70-452f-8b7b-4954fa7fce4c-kube-api-access-g4hmt\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361168 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361194 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361233 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.361738 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.361716 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.362301 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.362279 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.363683 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.362951 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.363683 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.363582 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.365448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.364141 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.370165 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.368439 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.373046 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.373007 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.425674 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.425646 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec12978d-ab70-452f-8b7b-4954fa7fce4c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.437114 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.436983 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hmt\" (UniqueName: \"kubernetes.io/projected/ec12978d-ab70-452f-8b7b-4954fa7fce4c-kube-api-access-g4hmt\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph\" (UID: \"ec12978d-ab70-452f-8b7b-4954fa7fce4c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.497904 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.497440 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:08.715500 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.715424 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z"] Oct 30 21:51:08.924000 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:08.923966 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph"] Oct 30 21:51:09.132929 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:09.132890 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" event={"ID":"ff7805e9-b57f-41cf-b098-6e6e28f2dc73","Type":"ContainerStarted","Data":"f7a22f081372a675b57aad130a90c5569cc26cc4a33b68328e42750e10599bd9"} Oct 30 21:51:09.138167 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:09.138135 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" event={"ID":"ec12978d-ab70-452f-8b7b-4954fa7fce4c","Type":"ContainerStarted","Data":"26a19c977bbd2a0b919394546f0e58b0c2981b0ffe86252d8f44913ad68cd54f"} Oct 30 21:51:17.926169 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:51:17.926091 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:51:18.280472 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:18.280383 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:51:18.280583 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:18.280492 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:51:18.280583 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:18.280539 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:51:18.307379 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:18.307149 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:51:18.307379 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:18.307239 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:51:18.307379 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:18.307350 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:51:19.232523 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.232479 4991 generic.go:356] "Generic (PLEG): container finished" podID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerID="3e03c355d4b6f3378d2a7d02fa80b4e2b34c857bedfa60c1bc45573b7da54c7c" exitCode=0 Oct 30 21:51:19.233120 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.232594 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" event={"ID":"f0d3536e-9ac4-48e1-8647-3370c950e0a3","Type":"ContainerDied","Data":"3e03c355d4b6f3378d2a7d02fa80b4e2b34c857bedfa60c1bc45573b7da54c7c"} Oct 30 21:51:19.236592 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.236521 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" event={"ID":"ec12978d-ab70-452f-8b7b-4954fa7fce4c","Type":"ContainerStarted","Data":"f5b8bf8482d2b986d4577ab6d98f10b81ee9c37e37417534776ffa41a5a31a94"} Oct 30 21:51:19.239862 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.239798 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" event={"ID":"ff7805e9-b57f-41cf-b098-6e6e28f2dc73","Type":"ContainerStarted","Data":"240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c"} Oct 30 21:51:19.460411 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.459771 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" podStartSLOduration=2.090337188 podStartE2EDuration="11.459753416s" totalImagesPullingTime="9.369416228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:19.458020799 +0000 UTC m=+299.757430713" watchObservedRunningTime="2026-10-30 21:51:19.459753416 +0000 UTC m=+299.759163299" Oct 30 21:51:19.498288 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.497948 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:19.504431 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.504404 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:19.573100 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.573046 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podStartSLOduration=3.020963105 podStartE2EDuration="12.573031036s" totalImagesPullingTime="9.552067931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:19.571727747 +0000 UTC m=+299.871137659" watchObservedRunningTime="2026-10-30 21:51:19.573031036 +0000 UTC m=+299.872440912" Oct 30 21:51:19.911765 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.911704 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:51:19.912123 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:19.912091 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:51:20.219270 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:20.219241 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:20.225455 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:20.225430 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:20.248638 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:20.248599 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" event={"ID":"d03f3aab-40d5-4ba7-b45a-2e7a334b0353","Type":"ContainerStarted","Data":"dd430124ce732968f52df2a7927be8d1421674d5fba63f5d22466ded6f68386d"} Oct 30 21:51:20.249254 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:20.249233 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:20.249460 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:20.249443 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:20.250081 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:20.250031 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:51:20.250678 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:20.250656 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph" Oct 30 21:51:21.259584 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.259555 4991 generic.go:356] "Generic (PLEG): container finished" podID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerID="dd430124ce732968f52df2a7927be8d1421674d5fba63f5d22466ded6f68386d" exitCode=0 Oct 30 21:51:21.259903 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.259676 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" event={"ID":"d03f3aab-40d5-4ba7-b45a-2e7a334b0353","Type":"ContainerDied","Data":"dd430124ce732968f52df2a7927be8d1421674d5fba63f5d22466ded6f68386d"} Oct 30 21:51:21.261504 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.261486 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:51:21.263332 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.263292 4991 generic.go:356] "Generic (PLEG): container finished" podID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerID="1504090dcf74308fd69117398326a2782a0bfa2fa2b12ec749cc0c81b4985ea1" exitCode=0 Oct 30 21:51:21.263649 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.263601 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" event={"ID":"2130981f-7ad9-49c5-9f44-b5d97c822b0c","Type":"ContainerDied","Data":"1504090dcf74308fd69117398326a2782a0bfa2fa2b12ec749cc0c81b4985ea1"} Oct 30 21:51:21.265926 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.265906 4991 generic.go:356] "Generic (PLEG): container finished" podID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerID="8d269fda63f2d6dff1db13732a8f61daf51f8b017b68b51d1ed65a28c1977ada" exitCode=0 Oct 30 21:51:21.266002 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.265982 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" event={"ID":"38ba858f-337c-4848-ae6f-80bbdaa19b12","Type":"ContainerDied","Data":"8d269fda63f2d6dff1db13732a8f61daf51f8b017b68b51d1ed65a28c1977ada"} Oct 30 21:51:21.267756 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.267734 4991 generic.go:356] "Generic (PLEG): container finished" podID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerID="6674be79c97c0f4b42bf2b88bde624b9deb19ff8f0e582f6069ce381d058a384" exitCode=0 Oct 30 21:51:21.267844 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:21.267823 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" event={"ID":"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc","Type":"ContainerDied","Data":"6674be79c97c0f4b42bf2b88bde624b9deb19ff8f0e582f6069ce381d058a384"} Oct 30 21:51:22.275759 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:22.275726 4991 generic.go:356] "Generic (PLEG): container finished" podID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerID="b8d390e660ad9b3e78c9cc60201be6e67bf3594fe34377b0114244f51479964b" exitCode=0 Oct 30 21:51:22.276185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:22.275815 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" event={"ID":"75a7f143-c4f5-42e6-8275-5c3d12c17b1e","Type":"ContainerDied","Data":"b8d390e660ad9b3e78c9cc60201be6e67bf3594fe34377b0114244f51479964b"} Oct 30 21:51:22.277857 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:22.277837 4991 generic.go:356] "Generic (PLEG): container finished" podID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerID="6ada37a700b0642672377b8135ba6cb54757fb3863149ea82712710b41aa4191" exitCode=0 Oct 30 21:51:22.278139 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:22.278100 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" event={"ID":"c3282554-842e-4103-a48c-b1c201a7ca9d","Type":"ContainerDied","Data":"6ada37a700b0642672377b8135ba6cb54757fb3863149ea82712710b41aa4191"} Oct 30 21:51:23.285760 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.285732 4991 generic.go:356] "Generic (PLEG): container finished" podID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerID="43e2a374bd8aef4081a007e523ef33aea726ba4a6d836034032a1fbbed00b9ba" exitCode=0 Oct 30 21:51:23.286076 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.285866 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" event={"ID":"d03f3aab-40d5-4ba7-b45a-2e7a334b0353","Type":"ContainerDied","Data":"43e2a374bd8aef4081a007e523ef33aea726ba4a6d836034032a1fbbed00b9ba"} Oct 30 21:51:23.288673 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.288200 4991 generic.go:356] "Generic (PLEG): container finished" podID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerID="edeacf05118147005b6968f9d55fcd68287b2500169e7c68d9e69da20cebe352" exitCode=0 Oct 30 21:51:23.288673 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.288282 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" event={"ID":"2130981f-7ad9-49c5-9f44-b5d97c822b0c","Type":"ContainerDied","Data":"edeacf05118147005b6968f9d55fcd68287b2500169e7c68d9e69da20cebe352"} Oct 30 21:51:23.290724 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.290658 4991 generic.go:356] "Generic (PLEG): container finished" podID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerID="a7aa25ab38981ea5c97be4ab46c0498ac21a14576c2f528eec2837283112699b" exitCode=0 Oct 30 21:51:23.290795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.290782 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" event={"ID":"38ba858f-337c-4848-ae6f-80bbdaa19b12","Type":"ContainerDied","Data":"a7aa25ab38981ea5c97be4ab46c0498ac21a14576c2f528eec2837283112699b"} Oct 30 21:51:23.292985 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.292874 4991 generic.go:356] "Generic (PLEG): container finished" podID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerID="a7aa9675b1b13bbaa95dad0179a4247d235a1ea821fe0407eb9a6eaa628b65fb" exitCode=0 Oct 30 21:51:23.292985 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:23.292903 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" event={"ID":"f0d3536e-9ac4-48e1-8647-3370c950e0a3","Type":"ContainerDied","Data":"a7aa9675b1b13bbaa95dad0179a4247d235a1ea821fe0407eb9a6eaa628b65fb"} Oct 30 21:51:24.300397 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:24.300347 4991 generic.go:356] "Generic (PLEG): container finished" podID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerID="198b4f79292945e4f1e198acc69cd5d5d7d55168bf6a6d1dd22d9c01a4a2dd74" exitCode=0 Oct 30 21:51:24.300864 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:24.300561 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" event={"ID":"75a7f143-c4f5-42e6-8275-5c3d12c17b1e","Type":"ContainerDied","Data":"198b4f79292945e4f1e198acc69cd5d5d7d55168bf6a6d1dd22d9c01a4a2dd74"} Oct 30 21:51:24.304234 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:24.304199 4991 generic.go:356] "Generic (PLEG): container finished" podID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerID="d0af66e36e45234787789c9669b8669742b21dd2829a5aefeb9de236836c8032" exitCode=0 Oct 30 21:51:24.304411 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:24.304265 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" event={"ID":"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc","Type":"ContainerDied","Data":"d0af66e36e45234787789c9669b8669742b21dd2829a5aefeb9de236836c8032"} Oct 30 21:51:27.337641 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:27.337443 4991 generic.go:356] "Generic (PLEG): container finished" podID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerID="a12a378ec3d25a3d49566d7a3ed244d2df03ec69195e122cee52d8bfbb740886" exitCode=0 Oct 30 21:51:27.337641 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:27.337535 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" event={"ID":"c3282554-842e-4103-a48c-b1c201a7ca9d","Type":"ContainerDied","Data":"a12a378ec3d25a3d49566d7a3ed244d2df03ec69195e122cee52d8bfbb740886"} Oct 30 21:51:38.460076 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:38.459609 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" event={"ID":"38ba858f-337c-4848-ae6f-80bbdaa19b12","Type":"ContainerStarted","Data":"aaa5894e0dc9f71e5249f29466d53fa87d8a8fa3965c76de7b6bbf0fc814e915"} Oct 30 21:51:38.462109 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:38.462085 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" event={"ID":"d03f3aab-40d5-4ba7-b45a-2e7a334b0353","Type":"ContainerStarted","Data":"78ae2d944f6368caf1f3474b607900dd0b928c6fafcbbd93698f2ecfe209fca1"} Oct 30 21:51:38.615475 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:38.615419 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" podStartSLOduration=3.3873372330000002 podStartE2EDuration="35.615398656s" totalImagesPullingTime="32.228061423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:38.613207371 +0000 UTC m=+318.912617268" watchObservedRunningTime="2026-10-30 21:51:38.615398656 +0000 UTC m=+318.914808537" Oct 30 21:51:38.725609 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:38.725539 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" podStartSLOduration=3.258310521 podStartE2EDuration="35.725519762s" totalImagesPullingTime="32.467209241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:38.723923885 +0000 UTC m=+319.023333765" watchObservedRunningTime="2026-10-30 21:51:38.725519762 +0000 UTC m=+319.024929643" Oct 30 21:51:39.469457 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.469424 4991 generic.go:356] "Generic (PLEG): container finished" podID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerID="aaa5894e0dc9f71e5249f29466d53fa87d8a8fa3965c76de7b6bbf0fc814e915" exitCode=0 Oct 30 21:51:39.469844 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.469511 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" event={"ID":"38ba858f-337c-4848-ae6f-80bbdaa19b12","Type":"ContainerDied","Data":"aaa5894e0dc9f71e5249f29466d53fa87d8a8fa3965c76de7b6bbf0fc814e915"} Oct 30 21:51:39.472860 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.472839 4991 generic.go:356] "Generic (PLEG): container finished" podID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerID="e25713d0f9890814a10a782688e92b09aab6390d5e9cf089b42c67286a43d708" exitCode=0 Oct 30 21:51:39.472962 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.472902 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" event={"ID":"f0d3536e-9ac4-48e1-8647-3370c950e0a3","Type":"ContainerDied","Data":"e25713d0f9890814a10a782688e92b09aab6390d5e9cf089b42c67286a43d708"} Oct 30 21:51:39.475685 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.475659 4991 generic.go:356] "Generic (PLEG): container finished" podID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerID="78ae2d944f6368caf1f3474b607900dd0b928c6fafcbbd93698f2ecfe209fca1" exitCode=0 Oct 30 21:51:39.475804 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.475750 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" event={"ID":"d03f3aab-40d5-4ba7-b45a-2e7a334b0353","Type":"ContainerDied","Data":"78ae2d944f6368caf1f3474b607900dd0b928c6fafcbbd93698f2ecfe209fca1"} Oct 30 21:51:39.478758 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.478737 4991 generic.go:356] "Generic (PLEG): container finished" podID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerID="f222b249d6afecbe0ebea83109cbdd5de693ea796c3f306ee7207b94e1df7a64" exitCode=0 Oct 30 21:51:39.478852 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:39.478827 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" event={"ID":"2130981f-7ad9-49c5-9f44-b5d97c822b0c","Type":"ContainerDied","Data":"f222b249d6afecbe0ebea83109cbdd5de693ea796c3f306ee7207b94e1df7a64"} Oct 30 21:51:40.487562 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.487518 4991 generic.go:356] "Generic (PLEG): container finished" podID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerID="efb4d26009d160bd430b5ce2396d9f45e2cc46ac6b7990650c8a85a4e458be0f" exitCode=0 Oct 30 21:51:40.487981 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.487602 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" event={"ID":"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc","Type":"ContainerDied","Data":"efb4d26009d160bd430b5ce2396d9f45e2cc46ac6b7990650c8a85a4e458be0f"} Oct 30 21:51:40.490600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.490577 4991 generic.go:356] "Generic (PLEG): container finished" podID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerID="f99f15d8a7dcdaa476c7883c9d6af534df2c227f4a240c6852d3c61c02d99eef" exitCode=0 Oct 30 21:51:40.490767 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.490611 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" event={"ID":"75a7f143-c4f5-42e6-8275-5c3d12c17b1e","Type":"ContainerDied","Data":"f99f15d8a7dcdaa476c7883c9d6af534df2c227f4a240c6852d3c61c02d99eef"} Oct 30 21:51:40.707567 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.707546 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:40.721219 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.721199 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:40.724531 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.724513 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:40.727715 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.727698 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:40.812079 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812006 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-bundle\") pod \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " Oct 30 21:51:40.812079 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812043 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-kube-api-access-vnnz6\" (UniqueName: \"kubernetes.io/projected/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-kube-api-access-vnnz6\") pod \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " Oct 30 21:51:40.812079 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812073 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-util\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-util\") pod \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " Oct 30 21:51:40.812367 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812147 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-bundle\") pod \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " Oct 30 21:51:40.812367 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812182 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2130981f-7ad9-49c5-9f44-b5d97c822b0c-kube-api-access-qlbzc\" (UniqueName: \"kubernetes.io/projected/2130981f-7ad9-49c5-9f44-b5d97c822b0c-kube-api-access-qlbzc\") pod \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " Oct 30 21:51:40.812367 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812218 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-util\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-util\") pod \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " Oct 30 21:51:40.812367 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812277 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ba858f-337c-4848-ae6f-80bbdaa19b12-kube-api-access-dvhxk\" (UniqueName: \"kubernetes.io/projected/38ba858f-337c-4848-ae6f-80bbdaa19b12-kube-api-access-dvhxk\") pod \"38ba858f-337c-4848-ae6f-80bbdaa19b12\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " Oct 30 21:51:40.812367 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812346 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-bundle\") pod \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\" (UID: \"2130981f-7ad9-49c5-9f44-b5d97c822b0c\") " Oct 30 21:51:40.812623 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812377 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-bundle\") pod \"38ba858f-337c-4848-ae6f-80bbdaa19b12\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " Oct 30 21:51:40.812623 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812401 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-util\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-util\") pod \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\" (UID: \"d03f3aab-40d5-4ba7-b45a-2e7a334b0353\") " Oct 30 21:51:40.812623 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812428 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-util\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-util\") pod \"38ba858f-337c-4848-ae6f-80bbdaa19b12\" (UID: \"38ba858f-337c-4848-ae6f-80bbdaa19b12\") " Oct 30 21:51:40.812623 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812487 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d3536e-9ac4-48e1-8647-3370c950e0a3-kube-api-access-pgkzw\" (UniqueName: \"kubernetes.io/projected/f0d3536e-9ac4-48e1-8647-3370c950e0a3-kube-api-access-pgkzw\") pod \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\" (UID: \"f0d3536e-9ac4-48e1-8647-3370c950e0a3\") " Oct 30 21:51:40.812623 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812553 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-bundle" pod "f0d3536e-9ac4-48e1-8647-3370c950e0a3" (UID: "f0d3536e-9ac4-48e1-8647-3370c950e0a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.812866 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812695 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-bundle" pod "d03f3aab-40d5-4ba7-b45a-2e7a334b0353" (UID: "d03f3aab-40d5-4ba7-b45a-2e7a334b0353"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.812866 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.812748 4991 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.813107 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.813082 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-bundle" pod "2130981f-7ad9-49c5-9f44-b5d97c822b0c" (UID: "2130981f-7ad9-49c5-9f44-b5d97c822b0c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.814843 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.814804 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-bundle" pod "38ba858f-337c-4848-ae6f-80bbdaa19b12" (UID: "38ba858f-337c-4848-ae6f-80bbdaa19b12"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.814933 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.814910 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-kube-api-access-vnnz6" pod "d03f3aab-40d5-4ba7-b45a-2e7a334b0353" (UID: "d03f3aab-40d5-4ba7-b45a-2e7a334b0353"). InnerVolumeSpecName "kube-api-access-vnnz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.814995 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.814923 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ba858f-337c-4848-ae6f-80bbdaa19b12-kube-api-access-dvhxk" pod "38ba858f-337c-4848-ae6f-80bbdaa19b12" (UID: "38ba858f-337c-4848-ae6f-80bbdaa19b12"). InnerVolumeSpecName "kube-api-access-dvhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.815396 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.815369 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2130981f-7ad9-49c5-9f44-b5d97c822b0c-kube-api-access-qlbzc" pod "2130981f-7ad9-49c5-9f44-b5d97c822b0c" (UID: "2130981f-7ad9-49c5-9f44-b5d97c822b0c"). InnerVolumeSpecName "kube-api-access-qlbzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.816906 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.816879 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d3536e-9ac4-48e1-8647-3370c950e0a3-kube-api-access-pgkzw" pod "f0d3536e-9ac4-48e1-8647-3370c950e0a3" (UID: "f0d3536e-9ac4-48e1-8647-3370c950e0a3"). InnerVolumeSpecName "kube-api-access-pgkzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.818572 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.818550 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-util" pod "2130981f-7ad9-49c5-9f44-b5d97c822b0c" (UID: "2130981f-7ad9-49c5-9f44-b5d97c822b0c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.819235 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.819209 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-util" pod "f0d3536e-9ac4-48e1-8647-3370c950e0a3" (UID: "f0d3536e-9ac4-48e1-8647-3370c950e0a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.819669 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.819650 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-util" pod "d03f3aab-40d5-4ba7-b45a-2e7a334b0353" (UID: "d03f3aab-40d5-4ba7-b45a-2e7a334b0353"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.819856 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.819837 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-util" pod "38ba858f-337c-4848-ae6f-80bbdaa19b12" (UID: "38ba858f-337c-4848-ae6f-80bbdaa19b12"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.913095 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913070 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnnz6\" (UniqueName: \"kubernetes.io/projected/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-kube-api-access-vnnz6\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913095 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913094 4991 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-util\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913108 4991 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913120 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlbzc\" (UniqueName: \"kubernetes.io/projected/2130981f-7ad9-49c5-9f44-b5d97c822b0c-kube-api-access-qlbzc\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913133 4991 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0d3536e-9ac4-48e1-8647-3370c950e0a3-util\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913147 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvhxk\" (UniqueName: \"kubernetes.io/projected/38ba858f-337c-4848-ae6f-80bbdaa19b12-kube-api-access-dvhxk\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913158 4991 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2130981f-7ad9-49c5-9f44-b5d97c822b0c-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913170 4991 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913182 4991 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d03f3aab-40d5-4ba7-b45a-2e7a334b0353-util\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913196 4991 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ba858f-337c-4848-ae6f-80bbdaa19b12-util\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.913273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:40.913208 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgkzw\" (UniqueName: \"kubernetes.io/projected/f0d3536e-9ac4-48e1-8647-3370c950e0a3-kube-api-access-pgkzw\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.496854 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.496832 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" Oct 30 21:51:41.497235 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.496832 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tpvfz" event={"ID":"2130981f-7ad9-49c5-9f44-b5d97c822b0c","Type":"ContainerDied","Data":"a5ecf93df8f83561f770925f6f1404668c9f2db91cb1cdc2d26a44ed2ae90e79"} Oct 30 21:51:41.497235 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.496953 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ecf93df8f83561f770925f6f1404668c9f2db91cb1cdc2d26a44ed2ae90e79" Oct 30 21:51:41.499100 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.499074 4991 generic.go:356] "Generic (PLEG): container finished" podID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerID="ae6eb59546cc9192cd6aaf84a2dc32ae546dbdbe5aa62cd519479b34fec0e2d5" exitCode=0 Oct 30 21:51:41.499226 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.499152 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" event={"ID":"c3282554-842e-4103-a48c-b1c201a7ca9d","Type":"ContainerDied","Data":"ae6eb59546cc9192cd6aaf84a2dc32ae546dbdbe5aa62cd519479b34fec0e2d5"} Oct 30 21:51:41.501198 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.501179 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" Oct 30 21:51:41.501198 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.501193 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zzlq" event={"ID":"38ba858f-337c-4848-ae6f-80bbdaa19b12","Type":"ContainerDied","Data":"59ff9a66ee6b4aa971923f3b547e16240359bf81efe95f775c566b5e3a4e7072"} Oct 30 21:51:41.501347 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.501224 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ff9a66ee6b4aa971923f3b547e16240359bf81efe95f775c566b5e3a4e7072" Oct 30 21:51:41.503200 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.503181 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" Oct 30 21:51:41.503312 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.503177 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dj2xw" event={"ID":"f0d3536e-9ac4-48e1-8647-3370c950e0a3","Type":"ContainerDied","Data":"7f9864d02152b68a43197ce230fc61c7c830fb83ac507ca4ff5205562baeca1c"} Oct 30 21:51:41.503312 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.503303 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9864d02152b68a43197ce230fc61c7c830fb83ac507ca4ff5205562baeca1c" Oct 30 21:51:41.505396 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.505341 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" event={"ID":"d03f3aab-40d5-4ba7-b45a-2e7a334b0353","Type":"ContainerDied","Data":"f93b0a2ef203a3a7fa3f01b914e742c8c75034b1c76bffaac0d184887e26c053"} Oct 30 21:51:41.505396 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.505364 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77lfkj" Oct 30 21:51:41.505396 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.505380 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93b0a2ef203a3a7fa3f01b914e742c8c75034b1c76bffaac0d184887e26c053" Oct 30 21:51:41.631576 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.631551 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:41.663656 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.663636 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:41.720302 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.720266 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-util\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-util\") pod \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " Oct 30 21:51:41.720423 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.720351 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-kube-api-access-vcnhd\" (UniqueName: \"kubernetes.io/projected/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-kube-api-access-vcnhd\") pod \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " Oct 30 21:51:41.720423 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.720411 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-util\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-util\") pod \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " Oct 30 21:51:41.720539 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.720458 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-bundle\") pod \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\" (UID: \"75a7f143-c4f5-42e6-8275-5c3d12c17b1e\") " Oct 30 21:51:41.720539 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.720529 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-kube-api-access-fxx9v\" (UniqueName: \"kubernetes.io/projected/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-kube-api-access-fxx9v\") pod \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " Oct 30 21:51:41.720631 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.720619 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-bundle\") pod \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\" (UID: \"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc\") " Oct 30 21:51:41.722060 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.721977 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-bundle" pod "60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" (UID: "60e030d2-d8cd-4dd2-87a2-1c47c4d414bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.722563 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.722528 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-bundle" pod "75a7f143-c4f5-42e6-8275-5c3d12c17b1e" (UID: "75a7f143-c4f5-42e6-8275-5c3d12c17b1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.727530 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.727500 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-kube-api-access-vcnhd" pod "75a7f143-c4f5-42e6-8275-5c3d12c17b1e" (UID: "75a7f143-c4f5-42e6-8275-5c3d12c17b1e"). InnerVolumeSpecName "kube-api-access-vcnhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:41.728913 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.728885 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-kube-api-access-fxx9v" pod "60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" (UID: "60e030d2-d8cd-4dd2-87a2-1c47c4d414bc"). InnerVolumeSpecName "kube-api-access-fxx9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:41.729196 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.729172 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-util" pod "60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" (UID: "60e030d2-d8cd-4dd2-87a2-1c47c4d414bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.731158 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.731133 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-util" pod "75a7f143-c4f5-42e6-8275-5c3d12c17b1e" (UID: "75a7f143-c4f5-42e6-8275-5c3d12c17b1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.821889 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.821869 4991 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.821889 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.821889 4991 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-util\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.822003 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.821898 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcnhd\" (UniqueName: \"kubernetes.io/projected/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-kube-api-access-vcnhd\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.822003 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.821906 4991 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-util\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.822003 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.821915 4991 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a7f143-c4f5-42e6-8275-5c3d12c17b1e-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.822003 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:41.821922 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxx9v\" (UniqueName: \"kubernetes.io/projected/60e030d2-d8cd-4dd2-87a2-1c47c4d414bc-kube-api-access-fxx9v\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:42.511719 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.511690 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" Oct 30 21:51:42.512155 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.511687 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49g9q" event={"ID":"60e030d2-d8cd-4dd2-87a2-1c47c4d414bc","Type":"ContainerDied","Data":"e8346058ac3d1840fb77c3a2f590141a727930630da2c45c656d788a4fa5299a"} Oct 30 21:51:42.512155 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.511791 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8346058ac3d1840fb77c3a2f590141a727930630da2c45c656d788a4fa5299a" Oct 30 21:51:42.513845 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.513810 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" event={"ID":"75a7f143-c4f5-42e6-8275-5c3d12c17b1e","Type":"ContainerDied","Data":"f77b8a4f1b7c2ff98eed77b9b89c36fc0509e9adb6afa738bb695f8837d73c4c"} Oct 30 21:51:42.514004 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.513989 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f77b8a4f1b7c2ff98eed77b9b89c36fc0509e9adb6afa738bb695f8837d73c4c" Oct 30 21:51:42.514107 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.514043 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdlxx" Oct 30 21:51:42.630771 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.630746 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:42.729053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.729025 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-util\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-util\") pod \"c3282554-842e-4103-a48c-b1c201a7ca9d\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " Oct 30 21:51:42.729230 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.729091 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3282554-842e-4103-a48c-b1c201a7ca9d-kube-api-access-wh2wg\" (UniqueName: \"kubernetes.io/projected/c3282554-842e-4103-a48c-b1c201a7ca9d-kube-api-access-wh2wg\") pod \"c3282554-842e-4103-a48c-b1c201a7ca9d\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " Oct 30 21:51:42.729230 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.729122 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-bundle\") pod \"c3282554-842e-4103-a48c-b1c201a7ca9d\" (UID: \"c3282554-842e-4103-a48c-b1c201a7ca9d\") " Oct 30 21:51:42.729834 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.729806 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-bundle" pod "c3282554-842e-4103-a48c-b1c201a7ca9d" (UID: "c3282554-842e-4103-a48c-b1c201a7ca9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:42.731199 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.731174 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3282554-842e-4103-a48c-b1c201a7ca9d-kube-api-access-wh2wg" pod "c3282554-842e-4103-a48c-b1c201a7ca9d" (UID: "c3282554-842e-4103-a48c-b1c201a7ca9d"). InnerVolumeSpecName "kube-api-access-wh2wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:42.734620 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.734598 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-util" pod "c3282554-842e-4103-a48c-b1c201a7ca9d" (UID: "c3282554-842e-4103-a48c-b1c201a7ca9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:42.830275 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.830218 4991 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-bundle\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:42.830275 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.830242 4991 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3282554-842e-4103-a48c-b1c201a7ca9d-util\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:42.830275 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:42.830252 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2wg\" (UniqueName: \"kubernetes.io/projected/c3282554-842e-4103-a48c-b1c201a7ca9d-kube-api-access-wh2wg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:43.520593 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:43.520560 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" event={"ID":"c3282554-842e-4103-a48c-b1c201a7ca9d","Type":"ContainerDied","Data":"bd3939c88db5290f598cbb04f551527a179eefb3151a559fd68d3fd0b453f9f7"} Oct 30 21:51:43.521055 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:43.520611 4991 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd3939c88db5290f598cbb04f551527a179eefb3151a559fd68d3fd0b453f9f7" Oct 30 21:51:43.521055 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:43.520571 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqzb8" Oct 30 21:51:47.908612 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:51:47.908579 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:51:55.080280 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080246 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn"] Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080574 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080586 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080601 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080606 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080614 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080619 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080625 4991 cpu_manager.go:414] "containerMap: removing container" podUID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080630 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080638 4991 cpu_manager.go:414] "containerMap: removing container" podUID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerName="util" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080642 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerName="util" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080649 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerName="util" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080653 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerName="util" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080659 4991 cpu_manager.go:414] "containerMap: removing container" podUID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080664 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080671 4991 cpu_manager.go:414] "containerMap: removing container" podUID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080676 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerName="pull" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080682 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080686 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080692 4991 cpu_manager.go:414] "containerMap: removing container" podUID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080698 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080707 4991 cpu_manager.go:414] "containerMap: removing container" podUID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerName="util" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080712 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerName="util" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080717 4991 cpu_manager.go:414] "containerMap: removing container" podUID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerName="extract" Oct 30 21:51:55.080713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080722 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080731 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerName="pull" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080735 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerName="pull" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080742 4991 cpu_manager.go:414] "containerMap: removing container" podUID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerName="pull" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080747 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerName="pull" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080755 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080760 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080765 4991 cpu_manager.go:414] "containerMap: removing container" podUID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerName="pull" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080769 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerName="pull" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080777 4991 cpu_manager.go:414] "containerMap: removing container" podUID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080782 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080787 4991 cpu_manager.go:414] "containerMap: removing container" podUID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080792 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080797 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080801 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080808 4991 cpu_manager.go:414] "containerMap: removing container" podUID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080812 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080818 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080822 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerName="util" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080867 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2130981f-7ad9-49c5-9f44-b5d97c822b0c" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080873 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ba858f-337c-4848-ae6f-80bbdaa19b12" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080879 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d3536e-9ac4-48e1-8647-3370c950e0a3" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080885 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e030d2-d8cd-4dd2-87a2-1c47c4d414bc" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080891 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03f3aab-40d5-4ba7-b45a-2e7a334b0353" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080898 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3282554-842e-4103-a48c-b1c201a7ca9d" containerName="extract" Oct 30 21:51:55.081404 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.080904 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a7f143-c4f5-42e6-8275-5c3d12c17b1e" containerName="extract" Oct 30 21:51:55.083302 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.083285 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" Oct 30 21:51:55.136229 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.136209 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:51:55.136370 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.136302 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:51:55.182450 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.182424 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:51:55.182563 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.182527 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nl6dq\"" Oct 30 21:51:55.184261 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.184238 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn"] Oct 30 21:51:55.226641 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.226618 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbdw\" (UniqueName: \"kubernetes.io/projected/0ec987ff-0828-4a26-8926-874d8a47cbc7-kube-api-access-4jbdw\") pod \"dns-operator-controller-manager-8557cbdb84-6c8sn\" (UID: \"0ec987ff-0828-4a26-8926-874d8a47cbc7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" Oct 30 21:51:55.327143 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.327122 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbdw\" (UniqueName: \"kubernetes.io/projected/0ec987ff-0828-4a26-8926-874d8a47cbc7-kube-api-access-4jbdw\") pod \"dns-operator-controller-manager-8557cbdb84-6c8sn\" (UID: \"0ec987ff-0828-4a26-8926-874d8a47cbc7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" Oct 30 21:51:55.386138 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.386085 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbdw\" (UniqueName: \"kubernetes.io/projected/0ec987ff-0828-4a26-8926-874d8a47cbc7-kube-api-access-4jbdw\") pod \"dns-operator-controller-manager-8557cbdb84-6c8sn\" (UID: \"0ec987ff-0828-4a26-8926-874d8a47cbc7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" Oct 30 21:51:55.394852 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.394833 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" Oct 30 21:51:55.767380 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:55.767350 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn"] Oct 30 21:51:55.775054 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:51:55.775026 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec987ff_0828_4a26_8926_874d8a47cbc7.slice/crio-f10489d7ee45acdf8159e4998c68bd036f43a42eae6385231d8653ce00e3b697 WatchSource:0}: Error finding container f10489d7ee45acdf8159e4998c68bd036f43a42eae6385231d8653ce00e3b697: Status 404 returned error can't find the container with id f10489d7ee45acdf8159e4998c68bd036f43a42eae6385231d8653ce00e3b697 Oct 30 21:51:56.577666 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:56.577631 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" event={"ID":"0ec987ff-0828-4a26-8926-874d8a47cbc7","Type":"ContainerStarted","Data":"f10489d7ee45acdf8159e4998c68bd036f43a42eae6385231d8653ce00e3b697"} Oct 30 21:51:59.595206 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.595173 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf"] Oct 30 21:51:59.598547 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.598528 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" Oct 30 21:51:59.598688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.598554 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" event={"ID":"0ec987ff-0828-4a26-8926-874d8a47cbc7","Type":"ContainerStarted","Data":"ca57b7c8692052e7074f8db1aa80f674a963d6b56d0dd417cd06dcec0157e9a1"} Oct 30 21:51:59.598688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.598636 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:51:59.648561 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.648527 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf"] Oct 30 21:51:59.663959 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.663925 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5b7m\" (UniqueName: \"kubernetes.io/projected/c98fb20a-6b22-400a-b901-1338958d0ec6-kube-api-access-t5b7m\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf\" (UID: \"c98fb20a-6b22-400a-b901-1338958d0ec6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:51:59.664113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.664023 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c98fb20a-6b22-400a-b901-1338958d0ec6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf\" (UID: \"c98fb20a-6b22-400a-b901-1338958d0ec6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:51:59.692476 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.692445 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:51:59.692612 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.692537 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:51:59.692612 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.692581 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tkb6d\"" Oct 30 21:51:59.708354 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.708290 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" podStartSLOduration=1.376928971 podStartE2EDuration="4.708278408s" totalImagesPullingTime="3.331349437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:59.707692234 +0000 UTC m=+340.007102114" watchObservedRunningTime="2026-10-30 21:51:59.708278408 +0000 UTC m=+340.007688285" Oct 30 21:51:59.765357 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.765311 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5b7m\" (UniqueName: \"kubernetes.io/projected/c98fb20a-6b22-400a-b901-1338958d0ec6-kube-api-access-t5b7m\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf\" (UID: \"c98fb20a-6b22-400a-b901-1338958d0ec6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:51:59.765456 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.765379 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c98fb20a-6b22-400a-b901-1338958d0ec6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf\" (UID: \"c98fb20a-6b22-400a-b901-1338958d0ec6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:51:59.765726 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.765708 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c98fb20a-6b22-400a-b901-1338958d0ec6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf\" (UID: \"c98fb20a-6b22-400a-b901-1338958d0ec6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:51:59.828736 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.828710 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5b7m\" (UniqueName: \"kubernetes.io/projected/c98fb20a-6b22-400a-b901-1338958d0ec6-kube-api-access-t5b7m\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf\" (UID: \"c98fb20a-6b22-400a-b901-1338958d0ec6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:51:59.910259 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:51:59.910206 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" Oct 30 21:52:00.336542 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.336511 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf"] Oct 30 21:52:00.343165 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:00.343136 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98fb20a_6b22_400a_b901_1338958d0ec6.slice/crio-c7b228768bde64df3060d843c5fc06ac9474e0a3a46e22755a28ed906ce008ae WatchSource:0}: Error finding container c7b228768bde64df3060d843c5fc06ac9474e0a3a46e22755a28ed906ce008ae: Status 404 returned error can't find the container with id c7b228768bde64df3060d843c5fc06ac9474e0a3a46e22755a28ed906ce008ae Oct 30 21:52:00.476939 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.476910 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw"] Oct 30 21:52:00.480426 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.480403 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" Oct 30 21:52:00.530761 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.530742 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xgb8x\"" Oct 30 21:52:00.540835 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.540814 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw"] Oct 30 21:52:00.572696 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.572668 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjsk\" (UniqueName: \"kubernetes.io/projected/98b55f48-d625-4a7f-bbd7-98e462d435b5-kube-api-access-zfjsk\") pod \"authorino-operator-7b7bb4bc77-zdbtw\" (UID: \"98b55f48-d625-4a7f-bbd7-98e462d435b5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" Oct 30 21:52:00.600948 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.600887 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" event={"ID":"c98fb20a-6b22-400a-b901-1338958d0ec6","Type":"ContainerStarted","Data":"c7b228768bde64df3060d843c5fc06ac9474e0a3a46e22755a28ed906ce008ae"} Oct 30 21:52:00.673522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.673498 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjsk\" (UniqueName: \"kubernetes.io/projected/98b55f48-d625-4a7f-bbd7-98e462d435b5-kube-api-access-zfjsk\") pod \"authorino-operator-7b7bb4bc77-zdbtw\" (UID: \"98b55f48-d625-4a7f-bbd7-98e462d435b5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" Oct 30 21:52:00.734733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.734706 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjsk\" (UniqueName: \"kubernetes.io/projected/98b55f48-d625-4a7f-bbd7-98e462d435b5-kube-api-access-zfjsk\") pod \"authorino-operator-7b7bb4bc77-zdbtw\" (UID: \"98b55f48-d625-4a7f-bbd7-98e462d435b5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" Oct 30 21:52:00.792257 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:00.792234 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" Oct 30 21:52:01.159780 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:01.159740 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw"] Oct 30 21:52:01.166510 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:01.166482 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b55f48_d625_4a7f_bbd7_98e462d435b5.slice/crio-5170ecb61735cc07775e858fae076361b9b8b6b10449929325b5940313d41470 WatchSource:0}: Error finding container 5170ecb61735cc07775e858fae076361b9b8b6b10449929325b5940313d41470: Status 404 returned error can't find the container with id 5170ecb61735cc07775e858fae076361b9b8b6b10449929325b5940313d41470 Oct 30 21:52:01.606286 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:01.606249 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" event={"ID":"98b55f48-d625-4a7f-bbd7-98e462d435b5","Type":"ContainerStarted","Data":"5170ecb61735cc07775e858fae076361b9b8b6b10449929325b5940313d41470"} Oct 30 21:52:03.616774 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.616737 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" event={"ID":"98b55f48-d625-4a7f-bbd7-98e462d435b5","Type":"ContainerStarted","Data":"a2bd764d2727816904f6bee373c4af209f933e4412ec355a4a2c0b10e862dbfa"} Oct 30 21:52:03.617233 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.616809 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" Oct 30 21:52:03.668683 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.668646 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2"] Oct 30 21:52:03.672260 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.672239 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.728973 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.728920 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" podStartSLOduration=1.44452797 podStartE2EDuration="3.728905618s" totalImagesPullingTime="2.284377648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:03.728186261 +0000 UTC m=+344.027596137" watchObservedRunningTime="2026-10-30 21:52:03.728905618 +0000 UTC m=+344.028315495" Oct 30 21:52:03.794964 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.794934 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:52:03.795106 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.795093 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:52:03.795395 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.795379 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:52:03.795395 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.795390 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:52:03.800740 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.800718 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a40a162-322e-48e8-a3dc-f99245f209cf-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.800834 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.800749 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.800834 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.800769 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.800834 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.800791 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a162-322e-48e8-a3dc-f99245f209cf-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.800940 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.800922 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dpwp\" (UniqueName: \"kubernetes.io/projected/2a40a162-322e-48e8-a3dc-f99245f209cf-kube-api-access-7dpwp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.800976 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.800962 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.825487 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.825459 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sgld6\"" Oct 30 21:52:03.825837 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.825807 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:52:03.828165 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.828144 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2"] Oct 30 21:52:03.901930 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.901863 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a40a162-322e-48e8-a3dc-f99245f209cf-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.901930 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.901920 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.902191 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.901950 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.902191 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.901985 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a162-322e-48e8-a3dc-f99245f209cf-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.902191 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.902040 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dpwp\" (UniqueName: \"kubernetes.io/projected/2a40a162-322e-48e8-a3dc-f99245f209cf-kube-api-access-7dpwp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.902191 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.902079 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.902668 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.902640 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a40a162-322e-48e8-a3dc-f99245f209cf-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.904965 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.904931 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.905194 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.905173 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a40a162-322e-48e8-a3dc-f99245f209cf-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.905236 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.905182 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.905314 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.905297 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a40a162-322e-48e8-a3dc-f99245f209cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.972783 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.972755 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dpwp\" (UniqueName: \"kubernetes.io/projected/2a40a162-322e-48e8-a3dc-f99245f209cf-kube-api-access-7dpwp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-p5km2\" (UID: \"2a40a162-322e-48e8-a3dc-f99245f209cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:03.988611 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:03.988552 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:04.484846 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:04.484813 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2"] Oct 30 21:52:04.491235 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:04.491209 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a40a162_322e_48e8_a3dc_f99245f209cf.slice/crio-43b0f36780b17e7a7f7428a0fcd2774b439b8e595bab985c6b3ef025100b19ce WatchSource:0}: Error finding container 43b0f36780b17e7a7f7428a0fcd2774b439b8e595bab985c6b3ef025100b19ce: Status 404 returned error can't find the container with id 43b0f36780b17e7a7f7428a0fcd2774b439b8e595bab985c6b3ef025100b19ce Oct 30 21:52:04.623064 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:04.623029 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" event={"ID":"2a40a162-322e-48e8-a3dc-f99245f209cf","Type":"ContainerStarted","Data":"43b0f36780b17e7a7f7428a0fcd2774b439b8e595bab985c6b3ef025100b19ce"} Oct 30 21:52:07.642665 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:07.642567 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" event={"ID":"c98fb20a-6b22-400a-b901-1338958d0ec6","Type":"ContainerStarted","Data":"2823ae728eeb2dab1413304130625da61c2a5785c7c0874e7b72d38ba6595ebb"} Oct 30 21:52:07.855917 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:07.855853 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lv6hf" podStartSLOduration=2.274214196 podStartE2EDuration="8.855831384s" totalImagesPullingTime="6.581617188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:07.807275388 +0000 UTC m=+348.106685269" watchObservedRunningTime="2026-10-30 21:52:07.855831384 +0000 UTC m=+348.155241268" Oct 30 21:52:08.381197 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.381163 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8"] Oct 30 21:52:08.386775 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.386751 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:08.439929 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.439868 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dhmw8\"" Oct 30 21:52:08.484465 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.484432 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8"] Oct 30 21:52:08.540533 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.540500 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7679f\" (UniqueName: \"kubernetes.io/projected/0bf5a041-29ee-46b9-88ae-a1bfd29d24c2-kube-api-access-7679f\") pod \"kuadrant-operator-controller-manager-678d87766f-rhmj8\" (UID: \"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:08.540697 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.540552 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bf5a041-29ee-46b9-88ae-a1bfd29d24c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rhmj8\" (UID: \"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:08.643018 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.642982 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bf5a041-29ee-46b9-88ae-a1bfd29d24c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rhmj8\" (UID: \"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:08.643476 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.643141 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7679f\" (UniqueName: \"kubernetes.io/projected/0bf5a041-29ee-46b9-88ae-a1bfd29d24c2-kube-api-access-7679f\") pod \"kuadrant-operator-controller-manager-678d87766f-rhmj8\" (UID: \"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:08.643476 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.643250 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bf5a041-29ee-46b9-88ae-a1bfd29d24c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rhmj8\" (UID: \"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:08.649385 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.649357 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" event={"ID":"2a40a162-322e-48e8-a3dc-f99245f209cf","Type":"ContainerStarted","Data":"1ed207be63129d8340789349779e61f8113a5d51a3c7605f48126ea0fc3c16a5"} Oct 30 21:52:08.706957 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.706887 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7679f\" (UniqueName: \"kubernetes.io/projected/0bf5a041-29ee-46b9-88ae-a1bfd29d24c2-kube-api-access-7679f\") pod \"kuadrant-operator-controller-manager-678d87766f-rhmj8\" (UID: \"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:08.999491 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:08.999455 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:09.425920 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:09.425847 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8"] Oct 30 21:52:09.432309 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:09.432283 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf5a041_29ee_46b9_88ae_a1bfd29d24c2.slice/crio-12aa9d778020c9f919fb34a858f8b22f5153ad1d45c9a8897c91c3e40789d25a WatchSource:0}: Error finding container 12aa9d778020c9f919fb34a858f8b22f5153ad1d45c9a8897c91c3e40789d25a: Status 404 returned error can't find the container with id 12aa9d778020c9f919fb34a858f8b22f5153ad1d45c9a8897c91c3e40789d25a Oct 30 21:52:09.655356 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:09.655306 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" event={"ID":"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2","Type":"ContainerStarted","Data":"12aa9d778020c9f919fb34a858f8b22f5153ad1d45c9a8897c91c3e40789d25a"} Oct 30 21:52:09.656896 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:09.656871 4991 generic.go:356] "Generic (PLEG): container finished" podID="2a40a162-322e-48e8-a3dc-f99245f209cf" containerID="1ed207be63129d8340789349779e61f8113a5d51a3c7605f48126ea0fc3c16a5" exitCode=0 Oct 30 21:52:09.657016 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:09.656956 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" event={"ID":"2a40a162-322e-48e8-a3dc-f99245f209cf","Type":"ContainerDied","Data":"1ed207be63129d8340789349779e61f8113a5d51a3c7605f48126ea0fc3c16a5"} Oct 30 21:52:10.603548 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:10.603517 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c8sn" Oct 30 21:52:11.669041 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:11.669006 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" event={"ID":"2a40a162-322e-48e8-a3dc-f99245f209cf","Type":"ContainerStarted","Data":"36dec5471248daeb217f0042e74f49dc63f06d5f27420d3056b94ff5b541697b"} Oct 30 21:52:11.669527 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:11.669091 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:11.958010 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:11.957954 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" podStartSLOduration=3.831422328 podStartE2EDuration="8.957936099s" totalImagesPullingTime="4.623794541s" totalInitContainerRuntime="502.71923ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:11.911281223 +0000 UTC m=+352.210691100" watchObservedRunningTime="2026-10-30 21:52:11.957936099 +0000 UTC m=+352.257345985" Oct 30 21:52:14.625831 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:14.625800 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdbtw" Oct 30 21:52:14.690368 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:14.690280 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" event={"ID":"0bf5a041-29ee-46b9-88ae-a1bfd29d24c2","Type":"ContainerStarted","Data":"7c3780e72b00bcf14f1039ed2a5e4ea30c1d8cd12238bd5accb42e26393f4d8f"} Oct 30 21:52:14.690485 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:14.690455 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:14.878947 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:14.878898 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" podStartSLOduration=1.912888401 podStartE2EDuration="6.878883747s" totalImagesPullingTime="4.965995346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:14.876695965 +0000 UTC m=+355.176105845" watchObservedRunningTime="2026-10-30 21:52:14.878883747 +0000 UTC m=+355.178293628" Oct 30 21:52:17.909079 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:52:17.909042 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:52:22.678229 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:22.678197 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-p5km2" Oct 30 21:52:25.697137 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:25.697110 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rhmj8" Oct 30 21:52:36.803691 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.801550 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz"] Oct 30 21:52:36.810507 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.810480 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861224 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861191 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861391 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861270 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0038133-9d86-4cf5-a75f-539470632d85-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861391 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861304 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0038133-9d86-4cf5-a75f-539470632d85-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861391 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861365 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnt45\" (UniqueName: \"kubernetes.io/projected/e0038133-9d86-4cf5-a75f-539470632d85-kube-api-access-gnt45\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861396 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861450 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861497 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0038133-9d86-4cf5-a75f-539470632d85-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861649 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861526 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.861649 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.861554 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.863978 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.863955 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz"] Oct 30 21:52:36.907633 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.907614 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k8lqh\"" Oct 30 21:52:36.962062 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962034 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0038133-9d86-4cf5-a75f-539470632d85-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962213 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962081 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962213 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962149 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962316 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962210 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962316 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962261 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0038133-9d86-4cf5-a75f-539470632d85-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962316 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962289 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0038133-9d86-4cf5-a75f-539470632d85-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962358 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnt45\" (UniqueName: \"kubernetes.io/projected/e0038133-9d86-4cf5-a75f-539470632d85-kube-api-access-gnt45\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962403 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962482 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962480 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962610 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962553 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962649 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962628 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.962830 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.962803 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.963036 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.963012 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.963443 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.963424 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0038133-9d86-4cf5-a75f-539470632d85-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.964562 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.964544 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0038133-9d86-4cf5-a75f-539470632d85-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:36.964914 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:36.964897 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0038133-9d86-4cf5-a75f-539470632d85-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:37.020038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.020001 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0038133-9d86-4cf5-a75f-539470632d85-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:37.020134 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.020111 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnt45\" (UniqueName: \"kubernetes.io/projected/e0038133-9d86-4cf5-a75f-539470632d85-kube-api-access-gnt45\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz\" (UID: \"e0038133-9d86-4cf5-a75f-539470632d85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:37.124218 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.124149 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:37.548371 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.548336 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz"] Oct 30 21:52:37.554180 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:37.554150 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0038133_9d86_4cf5_a75f_539470632d85.slice/crio-55e8280b8c2fc6835adc3ec64aa0df3fa32ce6f9e1937d8577a53d204918566f WatchSource:0}: Error finding container 55e8280b8c2fc6835adc3ec64aa0df3fa32ce6f9e1937d8577a53d204918566f: Status 404 returned error can't find the container with id 55e8280b8c2fc6835adc3ec64aa0df3fa32ce6f9e1937d8577a53d204918566f Oct 30 21:52:37.556440 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.556410 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:52:37.556518 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.556476 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:52:37.556518 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.556508 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:52:37.803375 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.803262 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" event={"ID":"e0038133-9d86-4cf5-a75f-539470632d85","Type":"ContainerStarted","Data":"1860ee5ba01ed004243dd7692df0d4bcf3f12c46084ad718d460238446d9ec44"} Oct 30 21:52:37.803375 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.803359 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" event={"ID":"e0038133-9d86-4cf5-a75f-539470632d85","Type":"ContainerStarted","Data":"55e8280b8c2fc6835adc3ec64aa0df3fa32ce6f9e1937d8577a53d204918566f"} Oct 30 21:52:37.920391 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:37.920338 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" podStartSLOduration=1.9203080670000001 podStartE2EDuration="1.920308067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:37.919142067 +0000 UTC m=+378.218551946" watchObservedRunningTime="2026-10-30 21:52:37.920308067 +0000 UTC m=+378.219718024" Oct 30 21:52:38.125085 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:38.125010 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:38.129715 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:38.129689 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:38.808812 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:38.808776 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:38.809519 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:38.809502 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz" Oct 30 21:52:39.101537 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.101464 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xrzqj"] Oct 30 21:52:39.167706 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.167683 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.176867 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.176839 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xrzqj"] Oct 30 21:52:39.180643 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.180622 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbj4l\" (UniqueName: \"kubernetes.io/projected/2761ac10-0e5b-4bbd-ba7e-1a8773360476-kube-api-access-cbj4l\") pod \"limitador-limitador-74dfcfb894-xrzqj\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.180749 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.180655 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2761ac10-0e5b-4bbd-ba7e-1a8773360476-config-file\") pod \"limitador-limitador-74dfcfb894-xrzqj\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.219043 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.219027 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rmcm6\"" Oct 30 21:52:39.219553 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.219534 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:52:39.281271 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.281246 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbj4l\" (UniqueName: \"kubernetes.io/projected/2761ac10-0e5b-4bbd-ba7e-1a8773360476-kube-api-access-cbj4l\") pod \"limitador-limitador-74dfcfb894-xrzqj\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.281369 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.281279 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2761ac10-0e5b-4bbd-ba7e-1a8773360476-config-file\") pod \"limitador-limitador-74dfcfb894-xrzqj\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.281836 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.281819 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2761ac10-0e5b-4bbd-ba7e-1a8773360476-config-file\") pod \"limitador-limitador-74dfcfb894-xrzqj\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.339251 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.339225 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbj4l\" (UniqueName: \"kubernetes.io/projected/2761ac10-0e5b-4bbd-ba7e-1a8773360476-kube-api-access-cbj4l\") pod \"limitador-limitador-74dfcfb894-xrzqj\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.357534 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.357483 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xrzqj"] Oct 30 21:52:39.499851 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.499823 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:39.890791 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:39.890763 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xrzqj"] Oct 30 21:52:39.896941 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:39.896919 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2761ac10_0e5b_4bbd_ba7e_1a8773360476.slice/crio-b647d6520e8fcb174c750794195a8dbb839333fe7805bd20da8714d8cf48a073 WatchSource:0}: Error finding container b647d6520e8fcb174c750794195a8dbb839333fe7805bd20da8714d8cf48a073: Status 404 returned error can't find the container with id b647d6520e8fcb174c750794195a8dbb839333fe7805bd20da8714d8cf48a073 Oct 30 21:52:40.656642 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.656606 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wc752"] Oct 30 21:52:40.660495 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.660473 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wc752" Oct 30 21:52:40.693838 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.693811 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl296\" (UniqueName: \"kubernetes.io/projected/bd8ada48-f747-4329-9ba0-095e70319859-kube-api-access-hl296\") pod \"authorino-57f7f8659f-wc752\" (UID: \"bd8ada48-f747-4329-9ba0-095e70319859\") " pod="kuadrant-system/authorino-57f7f8659f-wc752" Oct 30 21:52:40.713436 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.713414 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wc752"] Oct 30 21:52:40.757481 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.757456 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b7wsj\"" Oct 30 21:52:40.794852 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.794827 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl296\" (UniqueName: \"kubernetes.io/projected/bd8ada48-f747-4329-9ba0-095e70319859-kube-api-access-hl296\") pod \"authorino-57f7f8659f-wc752\" (UID: \"bd8ada48-f747-4329-9ba0-095e70319859\") " pod="kuadrant-system/authorino-57f7f8659f-wc752" Oct 30 21:52:40.818750 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.818721 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" event={"ID":"2761ac10-0e5b-4bbd-ba7e-1a8773360476","Type":"ContainerStarted","Data":"b647d6520e8fcb174c750794195a8dbb839333fe7805bd20da8714d8cf48a073"} Oct 30 21:52:40.852886 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.852862 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl296\" (UniqueName: \"kubernetes.io/projected/bd8ada48-f747-4329-9ba0-095e70319859-kube-api-access-hl296\") pod \"authorino-57f7f8659f-wc752\" (UID: \"bd8ada48-f747-4329-9ba0-095e70319859\") " pod="kuadrant-system/authorino-57f7f8659f-wc752" Oct 30 21:52:40.981735 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:40.981709 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wc752" Oct 30 21:52:41.346907 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:41.346832 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wc752"] Oct 30 21:52:41.352518 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:41.352479 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8ada48_f747_4329_9ba0_095e70319859.slice/crio-5fd5576f2d72eef28ce019077220a356ff76a577239949a50ba4fe1c954c6800 WatchSource:0}: Error finding container 5fd5576f2d72eef28ce019077220a356ff76a577239949a50ba4fe1c954c6800: Status 404 returned error can't find the container with id 5fd5576f2d72eef28ce019077220a356ff76a577239949a50ba4fe1c954c6800 Oct 30 21:52:41.826374 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:41.826313 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wc752" event={"ID":"bd8ada48-f747-4329-9ba0-095e70319859","Type":"ContainerStarted","Data":"5fd5576f2d72eef28ce019077220a356ff76a577239949a50ba4fe1c954c6800"} Oct 30 21:52:42.833633 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:42.833596 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" event={"ID":"2761ac10-0e5b-4bbd-ba7e-1a8773360476","Type":"ContainerStarted","Data":"7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18"} Oct 30 21:52:42.834108 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:42.833675 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:42.945106 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:42.945058 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" podStartSLOduration=1.861110531 podStartE2EDuration="3.94504402s" totalImagesPullingTime="2.083933489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:42.943500377 +0000 UTC m=+383.242910257" watchObservedRunningTime="2026-10-30 21:52:42.94504402 +0000 UTC m=+383.244453917" Oct 30 21:52:44.845438 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:44.845399 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wc752" event={"ID":"bd8ada48-f747-4329-9ba0-095e70319859","Type":"ContainerStarted","Data":"378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1"} Oct 30 21:52:44.953607 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:44.953557 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-wc752" podStartSLOduration=1.963297942 podStartE2EDuration="4.953544273s" totalImagesPullingTime="2.990246331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:44.951968668 +0000 UTC m=+385.251378548" watchObservedRunningTime="2026-10-30 21:52:44.953544273 +0000 UTC m=+385.252954152" Oct 30 21:52:47.507497 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.507466 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wc752"] Oct 30 21:52:47.507894 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.507617 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-wc752" podUID="bd8ada48-f747-4329-9ba0-095e70319859" containerName="authorino" containerID="cri-o://378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1" gracePeriod=30 Oct 30 21:52:47.753705 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.753679 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wc752" Oct 30 21:52:47.859546 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.859483 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8ada48-f747-4329-9ba0-095e70319859-kube-api-access-hl296\" (UniqueName: \"kubernetes.io/projected/bd8ada48-f747-4329-9ba0-095e70319859-kube-api-access-hl296\") pod \"bd8ada48-f747-4329-9ba0-095e70319859\" (UID: \"bd8ada48-f747-4329-9ba0-095e70319859\") " Oct 30 21:52:47.861994 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.861969 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8ada48-f747-4329-9ba0-095e70319859-kube-api-access-hl296" pod "bd8ada48-f747-4329-9ba0-095e70319859" (UID: "bd8ada48-f747-4329-9ba0-095e70319859"). InnerVolumeSpecName "kube-api-access-hl296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:47.863116 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.863086 4991 generic.go:356] "Generic (PLEG): container finished" podID="bd8ada48-f747-4329-9ba0-095e70319859" containerID="378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1" exitCode=0 Oct 30 21:52:47.863612 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.863236 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wc752" event={"ID":"bd8ada48-f747-4329-9ba0-095e70319859","Type":"ContainerDied","Data":"378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1"} Oct 30 21:52:47.863612 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.863273 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wc752" event={"ID":"bd8ada48-f747-4329-9ba0-095e70319859","Type":"ContainerDied","Data":"5fd5576f2d72eef28ce019077220a356ff76a577239949a50ba4fe1c954c6800"} Oct 30 21:52:47.863612 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.863293 4991 scope.go:122] "RemoveContainer" containerID="378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1" Oct 30 21:52:47.863612 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.863565 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wc752" Oct 30 21:52:47.878811 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.878784 4991 scope.go:122] "RemoveContainer" containerID="378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1" Oct 30 21:52:47.879100 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:52:47.879080 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1\": container with ID starting with 378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1 not found: ID does not exist" containerID="378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1" Oct 30 21:52:47.879170 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.879114 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1"} err="failed to get container status \"378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1\": rpc error: code = NotFound desc = could not find container \"378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1\": container with ID starting with 378dec04de2b2413b6dc55db7e0f1b99c646f0b3c9a98f88260757e5cae826e1 not found: ID does not exist" Oct 30 21:52:47.909131 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:52:47.909099 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:52:47.960666 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:47.960634 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl296\" (UniqueName: \"kubernetes.io/projected/bd8ada48-f747-4329-9ba0-095e70319859-kube-api-access-hl296\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:48.026594 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:48.026569 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wc752"] Oct 30 21:52:48.030443 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:48.030420 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wc752"] Oct 30 21:52:49.873668 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:49.873627 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8ada48-f747-4329-9ba0-095e70319859" path="/var/lib/kubelet/pods/bd8ada48-f747-4329-9ba0-095e70319859/volumes" Oct 30 21:52:52.669434 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.669397 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm"] Oct 30 21:52:52.669798 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.669775 4991 cpu_manager.go:414] "containerMap: removing container" podUID="bd8ada48-f747-4329-9ba0-095e70319859" containerName="authorino" Oct 30 21:52:52.669798 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.669787 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8ada48-f747-4329-9ba0-095e70319859" containerName="authorino" Oct 30 21:52:52.669881 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.669857 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8ada48-f747-4329-9ba0-095e70319859" containerName="authorino" Oct 30 21:52:52.672635 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.672617 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.726527 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.726501 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 21:52:52.772628 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.772605 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-c8p94\"" Oct 30 21:52:52.772772 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.772675 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 21:52:52.772772 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.772731 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 21:52:52.773047 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.773033 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:52:52.775359 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.775339 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm"] Oct 30 21:52:52.802053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.802028 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fae213-5095-4bb1-a142-7c4a85cab98c-metrics-certs\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.802141 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.802061 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7fae213-5095-4bb1-a142-7c4a85cab98c-cert\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.802141 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.802120 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rxc\" (UniqueName: \"kubernetes.io/projected/a7fae213-5095-4bb1-a142-7c4a85cab98c-kube-api-access-v2rxc\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.802215 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.802148 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7fae213-5095-4bb1-a142-7c4a85cab98c-manager-config\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.832459 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.832439 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:52:52.902965 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.902942 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fae213-5095-4bb1-a142-7c4a85cab98c-metrics-certs\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.903087 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.902974 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7fae213-5095-4bb1-a142-7c4a85cab98c-cert\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.903087 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.903000 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rxc\" (UniqueName: \"kubernetes.io/projected/a7fae213-5095-4bb1-a142-7c4a85cab98c-kube-api-access-v2rxc\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.903087 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.903028 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7fae213-5095-4bb1-a142-7c4a85cab98c-manager-config\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.903814 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.903785 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7fae213-5095-4bb1-a142-7c4a85cab98c-manager-config\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.905426 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.905408 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7fae213-5095-4bb1-a142-7c4a85cab98c-cert\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.905517 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.905476 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fae213-5095-4bb1-a142-7c4a85cab98c-metrics-certs\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.960931 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.960879 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rxc\" (UniqueName: \"kubernetes.io/projected/a7fae213-5095-4bb1-a142-7c4a85cab98c-kube-api-access-v2rxc\") pod \"jobset-controller-manager-54489f778c-7dqqm\" (UID: \"a7fae213-5095-4bb1-a142-7c4a85cab98c\") " pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:52.983415 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:52.983396 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:53.356705 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:53.356666 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm"] Oct 30 21:52:53.362429 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:52:53.362395 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fae213_5095_4bb1_a142_7c4a85cab98c.slice/crio-e009f7495de10a5e10b79cf6a58813726f30e988a681de496fde59028b231f46 WatchSource:0}: Error finding container e009f7495de10a5e10b79cf6a58813726f30e988a681de496fde59028b231f46: Status 404 returned error can't find the container with id e009f7495de10a5e10b79cf6a58813726f30e988a681de496fde59028b231f46 Oct 30 21:52:53.840543 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:53.840507 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:53.893310 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:53.893284 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" event={"ID":"a7fae213-5095-4bb1-a142-7c4a85cab98c","Type":"ContainerStarted","Data":"e009f7495de10a5e10b79cf6a58813726f30e988a681de496fde59028b231f46"} Oct 30 21:52:57.914370 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:57.914307 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" event={"ID":"a7fae213-5095-4bb1-a142-7c4a85cab98c","Type":"ContainerStarted","Data":"fef629200e26576a0e79fe7c677a5259d5900673ac31500db81d806c52e68c01"} Oct 30 21:52:57.914745 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:57.914406 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:52:58.042194 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:58.042146 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" podStartSLOduration=2.2740131740000002 podStartE2EDuration="6.042132919s" totalImagesPullingTime="3.768119745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:58.035258593 +0000 UTC m=+398.334668494" watchObservedRunningTime="2026-10-30 21:52:58.042132919 +0000 UTC m=+398.341542800" Oct 30 21:52:59.291481 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.291442 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xrzqj"] Oct 30 21:52:59.291929 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.291665 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" podUID="2761ac10-0e5b-4bbd-ba7e-1a8773360476" containerName="limitador" containerID="cri-o://7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18" gracePeriod=30 Oct 30 21:52:59.838380 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.838349 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:59.924629 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.924567 4991 generic.go:356] "Generic (PLEG): container finished" podID="2761ac10-0e5b-4bbd-ba7e-1a8773360476" containerID="7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18" exitCode=0 Oct 30 21:52:59.924629 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.924612 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" Oct 30 21:52:59.924765 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.924651 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" event={"ID":"2761ac10-0e5b-4bbd-ba7e-1a8773360476","Type":"ContainerDied","Data":"7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18"} Oct 30 21:52:59.924765 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.924705 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xrzqj" event={"ID":"2761ac10-0e5b-4bbd-ba7e-1a8773360476","Type":"ContainerDied","Data":"b647d6520e8fcb174c750794195a8dbb839333fe7805bd20da8714d8cf48a073"} Oct 30 21:52:59.924765 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.924730 4991 scope.go:122] "RemoveContainer" containerID="7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18" Oct 30 21:52:59.933032 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.933016 4991 scope.go:122] "RemoveContainer" containerID="7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18" Oct 30 21:52:59.933296 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:52:59.933279 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18\": container with ID starting with 7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18 not found: ID does not exist" containerID="7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18" Oct 30 21:52:59.933363 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.933305 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18"} err="failed to get container status \"7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18\": rpc error: code = NotFound desc = could not find container \"7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18\": container with ID starting with 7c9a5e2a04ecd3dc28260e6c37424f5bc6e2379e4279ecbd0c50d32452535d18 not found: ID does not exist" Oct 30 21:52:59.964741 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.964718 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2761ac10-0e5b-4bbd-ba7e-1a8773360476-kube-api-access-cbj4l\" (UniqueName: \"kubernetes.io/projected/2761ac10-0e5b-4bbd-ba7e-1a8773360476-kube-api-access-cbj4l\") pod \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " Oct 30 21:52:59.964850 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.964757 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2761ac10-0e5b-4bbd-ba7e-1a8773360476-config-file\" (UniqueName: \"kubernetes.io/configmap/2761ac10-0e5b-4bbd-ba7e-1a8773360476-config-file\") pod \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\" (UID: \"2761ac10-0e5b-4bbd-ba7e-1a8773360476\") " Oct 30 21:52:59.965196 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.965169 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2761ac10-0e5b-4bbd-ba7e-1a8773360476-config-file" pod "2761ac10-0e5b-4bbd-ba7e-1a8773360476" (UID: "2761ac10-0e5b-4bbd-ba7e-1a8773360476"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:59.966698 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:52:59.966673 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2761ac10-0e5b-4bbd-ba7e-1a8773360476-kube-api-access-cbj4l" pod "2761ac10-0e5b-4bbd-ba7e-1a8773360476" (UID: "2761ac10-0e5b-4bbd-ba7e-1a8773360476"). InnerVolumeSpecName "kube-api-access-cbj4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:00.066029 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:00.066005 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbj4l\" (UniqueName: \"kubernetes.io/projected/2761ac10-0e5b-4bbd-ba7e-1a8773360476-kube-api-access-cbj4l\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:00.066029 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:00.066028 4991 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2761ac10-0e5b-4bbd-ba7e-1a8773360476-config-file\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:00.338591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:00.338562 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xrzqj"] Oct 30 21:53:00.385298 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:00.385263 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xrzqj"] Oct 30 21:53:01.872908 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:01.872875 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2761ac10-0e5b-4bbd-ba7e-1a8773360476" path="/var/lib/kubelet/pods/2761ac10-0e5b-4bbd-ba7e-1a8773360476/volumes" Oct 30 21:53:08.924008 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:08.923976 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-54489f778c-7dqqm" Oct 30 21:53:17.909596 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:53:17.909558 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:53:28.890774 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.890735 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9f487f56-qj5rp"] Oct 30 21:53:28.891664 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.891249 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2761ac10-0e5b-4bbd-ba7e-1a8773360476" containerName="limitador" Oct 30 21:53:28.891664 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.891268 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2761ac10-0e5b-4bbd-ba7e-1a8773360476" containerName="limitador" Oct 30 21:53:28.891664 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.891434 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2761ac10-0e5b-4bbd-ba7e-1a8773360476" containerName="limitador" Oct 30 21:53:28.895998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.895976 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:28.980343 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.980308 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:53:28.989713 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.989683 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hws\" (UniqueName: \"kubernetes.io/projected/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-kube-api-access-24hws\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:28.989852 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.989826 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-webhook-cert\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:28.989922 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.989870 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-metrics-certs\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:28.992366 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.992212 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:53:28.992366 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.992301 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:53:28.992515 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.992404 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dq296\"" Oct 30 21:53:28.992515 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.992414 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:53:28.992515 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.992423 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:53:28.994651 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:28.994632 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9f487f56-qj5rp"] Oct 30 21:53:29.090439 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.090408 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-metrics-certs\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:29.090591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.090479 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hws\" (UniqueName: \"kubernetes.io/projected/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-kube-api-access-24hws\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:29.090591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.090521 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-webhook-cert\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:29.092799 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.092769 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-metrics-certs\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:29.092799 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.092788 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-webhook-cert\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:29.147956 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.147895 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hws\" (UniqueName: \"kubernetes.io/projected/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-kube-api-access-24hws\") pod \"maas-controller-7f9f487f56-qj5rp\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:29.209940 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.209916 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:29.489053 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.489027 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-569467969-472sq"] Oct 30 21:53:29.494808 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.494791 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.550140 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.550114 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569467969-472sq"] Oct 30 21:53:29.598274 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.598248 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn86j\" (UniqueName: \"kubernetes.io/projected/731fd329-f519-4400-af1b-dfdf91e13693-kube-api-access-cn86j\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.598427 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.598297 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731fd329-f519-4400-af1b-dfdf91e13693-metrics-certs\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.598427 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.598346 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/731fd329-f519-4400-af1b-dfdf91e13693-webhook-cert\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.615247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.615222 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9f487f56-qj5rp"] Oct 30 21:53:29.621960 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:53:29.621938 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dd50b5_76e6_4a2a_8a96_ffa9113e3c9b.slice/crio-36a249a1c6f142919dfb0d841b95807bbba8a9b975a8143d613ce0644bb1deda WatchSource:0}: Error finding container 36a249a1c6f142919dfb0d841b95807bbba8a9b975a8143d613ce0644bb1deda: Status 404 returned error can't find the container with id 36a249a1c6f142919dfb0d841b95807bbba8a9b975a8143d613ce0644bb1deda Oct 30 21:53:29.699859 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.699830 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/731fd329-f519-4400-af1b-dfdf91e13693-webhook-cert\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.699966 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.699897 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn86j\" (UniqueName: \"kubernetes.io/projected/731fd329-f519-4400-af1b-dfdf91e13693-kube-api-access-cn86j\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.700027 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.699968 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731fd329-f519-4400-af1b-dfdf91e13693-metrics-certs\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.702185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.702161 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731fd329-f519-4400-af1b-dfdf91e13693-metrics-certs\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.702268 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.702163 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/731fd329-f519-4400-af1b-dfdf91e13693-webhook-cert\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.757872 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.757824 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn86j\" (UniqueName: \"kubernetes.io/projected/731fd329-f519-4400-af1b-dfdf91e13693-kube-api-access-cn86j\") pod \"maas-controller-569467969-472sq\" (UID: \"731fd329-f519-4400-af1b-dfdf91e13693\") " pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:29.809055 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:29.809034 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:30.071663 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:30.071595 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" event={"ID":"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b","Type":"ContainerStarted","Data":"36a249a1c6f142919dfb0d841b95807bbba8a9b975a8143d613ce0644bb1deda"} Oct 30 21:53:30.179153 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:30.179121 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569467969-472sq"] Oct 30 21:53:30.184817 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:53:30.184791 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731fd329_f519_4400_af1b_dfdf91e13693.slice/crio-1541839f03c7f7241824bcc996dacde223febfa62edc689d7150d374e17acc92 WatchSource:0}: Error finding container 1541839f03c7f7241824bcc996dacde223febfa62edc689d7150d374e17acc92: Status 404 returned error can't find the container with id 1541839f03c7f7241824bcc996dacde223febfa62edc689d7150d374e17acc92 Oct 30 21:53:31.077555 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:31.077516 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569467969-472sq" event={"ID":"731fd329-f519-4400-af1b-dfdf91e13693","Type":"ContainerStarted","Data":"1541839f03c7f7241824bcc996dacde223febfa62edc689d7150d374e17acc92"} Oct 30 21:53:35.098306 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:35.098263 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" event={"ID":"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b","Type":"ContainerStarted","Data":"a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688"} Oct 30 21:53:35.098869 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:35.098371 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:35.099770 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:35.099748 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569467969-472sq" event={"ID":"731fd329-f519-4400-af1b-dfdf91e13693","Type":"ContainerStarted","Data":"028a6b5bce1f5bb1da95d35e2922a9e9575af989daa1d27aabe95b99587d733c"} Oct 30 21:53:35.099850 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:35.099800 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:35.219780 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:35.219732 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" podStartSLOduration=2.687895839 podStartE2EDuration="7.219718788s" totalImagesPullingTime="4.531822949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:35.218707661 +0000 UTC m=+435.518117541" watchObservedRunningTime="2026-10-30 21:53:35.219718788 +0000 UTC m=+435.519128692" Oct 30 21:53:35.334041 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:35.333981 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-569467969-472sq" podStartSLOduration=2.355787301 podStartE2EDuration="6.333966508s" totalImagesPullingTime="3.978179207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:35.331616096 +0000 UTC m=+435.631026002" watchObservedRunningTime="2026-10-30 21:53:35.333966508 +0000 UTC m=+435.633376425" Oct 30 21:53:46.106378 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.106331 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-569467969-472sq" Oct 30 21:53:46.106739 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.106415 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:46.269431 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.269399 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9f487f56-qj5rp"] Oct 30 21:53:46.269656 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.269616 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" podUID="e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" containerName="manager" containerID="cri-o://a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688" gracePeriod=30 Oct 30 21:53:46.504338 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.504298 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:46.535522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.535499 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-kube-api-access-24hws\" (UniqueName: \"kubernetes.io/projected/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-kube-api-access-24hws\") pod \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " Oct 30 21:53:46.535622 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.535535 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-webhook-cert\") pod \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " Oct 30 21:53:46.535622 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.535578 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-metrics-certs\") pod \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\" (UID: \"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b\") " Oct 30 21:53:46.537688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.537656 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-webhook-cert" pod "e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" (UID: "e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:46.537778 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.537714 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-kube-api-access-24hws" pod "e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" (UID: "e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b"). InnerVolumeSpecName "kube-api-access-24hws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:46.537817 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.537790 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-metrics-certs" pod "e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" (UID: "e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:46.636169 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.636136 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24hws\" (UniqueName: \"kubernetes.io/projected/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-kube-api-access-24hws\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:46.636169 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.636165 4991 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-webhook-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:46.636407 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:46.636178 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:47.159073 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.159038 4991 generic.go:356] "Generic (PLEG): container finished" podID="e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" containerID="a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688" exitCode=0 Oct 30 21:53:47.159467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.159082 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" Oct 30 21:53:47.159467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.159084 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" event={"ID":"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b","Type":"ContainerDied","Data":"a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688"} Oct 30 21:53:47.159467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.159218 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f487f56-qj5rp" event={"ID":"e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b","Type":"ContainerDied","Data":"36a249a1c6f142919dfb0d841b95807bbba8a9b975a8143d613ce0644bb1deda"} Oct 30 21:53:47.159467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.159247 4991 scope.go:122] "RemoveContainer" containerID="a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688" Oct 30 21:53:47.168211 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.168195 4991 scope.go:122] "RemoveContainer" containerID="a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688" Oct 30 21:53:47.168472 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:53:47.168453 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688\": container with ID starting with a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688 not found: ID does not exist" containerID="a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688" Oct 30 21:53:47.168551 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.168484 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688"} err="failed to get container status \"a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688\": rpc error: code = NotFound desc = could not find container \"a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688\": container with ID starting with a2eb3858cf34657d76c5d4d2d4b96c05db2573e20f045b3d058b3047e9e4d688 not found: ID does not exist" Oct 30 21:53:47.315372 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.315338 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9f487f56-qj5rp"] Oct 30 21:53:47.361946 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.361916 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f9f487f56-qj5rp"] Oct 30 21:53:47.873445 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:53:47.873415 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" path="/var/lib/kubelet/pods/e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b/volumes" Oct 30 21:53:47.909369 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:53:47.909334 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:54:07.616424 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.616392 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9zdg4"] Oct 30 21:54:07.616894 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.616769 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" containerName="manager" Oct 30 21:54:07.616894 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.616780 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" containerName="manager" Oct 30 21:54:07.616894 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.616851 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dd50b5-76e6-4a2a-8a96-ffa9113e3c9b" containerName="manager" Oct 30 21:54:07.623464 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.623443 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:07.677183 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.677154 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fb5ct\"" Oct 30 21:54:07.680220 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.680195 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9zdg4"] Oct 30 21:54:07.708029 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.707997 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a108075d-1569-471d-9f71-287c7d156af2-tls-certs\") pod \"model-serving-api-75bd89fdfc-9zdg4\" (UID: \"a108075d-1569-471d-9f71-287c7d156af2\") " pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:07.708135 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.708036 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrk6\" (UniqueName: \"kubernetes.io/projected/a108075d-1569-471d-9f71-287c7d156af2-kube-api-access-rxrk6\") pod \"model-serving-api-75bd89fdfc-9zdg4\" (UID: \"a108075d-1569-471d-9f71-287c7d156af2\") " pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:07.723759 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.723737 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:54:07.808837 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.808813 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a108075d-1569-471d-9f71-287c7d156af2-tls-certs\") pod \"model-serving-api-75bd89fdfc-9zdg4\" (UID: \"a108075d-1569-471d-9f71-287c7d156af2\") " pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:07.808949 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.808842 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrk6\" (UniqueName: \"kubernetes.io/projected/a108075d-1569-471d-9f71-287c7d156af2-kube-api-access-rxrk6\") pod \"model-serving-api-75bd89fdfc-9zdg4\" (UID: \"a108075d-1569-471d-9f71-287c7d156af2\") " pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:07.811296 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.811272 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a108075d-1569-471d-9f71-287c7d156af2-tls-certs\") pod \"model-serving-api-75bd89fdfc-9zdg4\" (UID: \"a108075d-1569-471d-9f71-287c7d156af2\") " pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:07.866845 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.866778 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrk6\" (UniqueName: \"kubernetes.io/projected/a108075d-1569-471d-9f71-287c7d156af2-kube-api-access-rxrk6\") pod \"model-serving-api-75bd89fdfc-9zdg4\" (UID: \"a108075d-1569-471d-9f71-287c7d156af2\") " pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:07.936031 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:07.936005 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:08.305647 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:08.305610 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9zdg4"] Oct 30 21:54:08.311384 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:54:08.311354 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda108075d_1569_471d_9f71_287c7d156af2.slice/crio-eee9db22ac3e6cd70388149d7afe7c39bed5bb055e3dbee1fa0885521d9d9b25 WatchSource:0}: Error finding container eee9db22ac3e6cd70388149d7afe7c39bed5bb055e3dbee1fa0885521d9d9b25: Status 404 returned error can't find the container with id eee9db22ac3e6cd70388149d7afe7c39bed5bb055e3dbee1fa0885521d9d9b25 Oct 30 21:54:09.271965 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:09.271929 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" event={"ID":"a108075d-1569-471d-9f71-287c7d156af2","Type":"ContainerStarted","Data":"eee9db22ac3e6cd70388149d7afe7c39bed5bb055e3dbee1fa0885521d9d9b25"} Oct 30 21:54:10.975258 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:10.975218 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr"] Oct 30 21:54:10.978972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:10.978951 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.028071 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.028049 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:54:11.028303 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.028287 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:11.037151 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.037125 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-openshift-service-ca\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.037251 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.037216 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgc6\" (UniqueName: \"kubernetes.io/projected/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-kube-api-access-wtgc6\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.037302 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.037269 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-metrics-certs\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.037444 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.037421 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-maas-api-tls\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.074781 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.074755 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:11.074943 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.074926 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwpgv\"" Oct 30 21:54:11.074994 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.074939 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:54:11.117219 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.117189 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr"] Oct 30 21:54:11.138218 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.138191 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-maas-api-tls\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.138218 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.138218 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-openshift-service-ca\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.138410 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.138270 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgc6\" (UniqueName: \"kubernetes.io/projected/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-kube-api-access-wtgc6\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.138410 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.138297 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-metrics-certs\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.138971 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.138948 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-openshift-service-ca\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.140787 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.140765 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-metrics-certs\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.140871 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.140806 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-maas-api-tls\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.198718 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.198698 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgc6\" (UniqueName: \"kubernetes.io/projected/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-kube-api-access-wtgc6\") pod \"maas-api-59c9ddfb7b-86xvr\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.284769 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.284690 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" event={"ID":"a108075d-1569-471d-9f71-287c7d156af2","Type":"ContainerStarted","Data":"980a7bf0f6ec8bb2167934ce78061c8a988b31f8da11324413f632b607d48695"} Oct 30 21:54:11.284920 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.284817 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:11.294088 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.294065 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:11.395997 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.395945 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" podStartSLOduration=2.104330123 podStartE2EDuration="4.39593052s" totalImagesPullingTime="2.291600397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:11.39394554 +0000 UTC m=+471.693355420" watchObservedRunningTime="2026-10-30 21:54:11.39593052 +0000 UTC m=+471.695340399" Oct 30 21:54:11.715582 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:11.715512 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr"] Oct 30 21:54:11.721105 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:54:11.721074 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc548e5b4_d7c1_4a9e_88cd_17f5f7e6fdfe.slice/crio-55c65d54bf01827ce6039cb95df425c70a2263d6c7b3fa6f576e676bee0bfc49 WatchSource:0}: Error finding container 55c65d54bf01827ce6039cb95df425c70a2263d6c7b3fa6f576e676bee0bfc49: Status 404 returned error can't find the container with id 55c65d54bf01827ce6039cb95df425c70a2263d6c7b3fa6f576e676bee0bfc49 Oct 30 21:54:12.291337 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:12.291287 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" event={"ID":"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe","Type":"ContainerStarted","Data":"55c65d54bf01827ce6039cb95df425c70a2263d6c7b3fa6f576e676bee0bfc49"} Oct 30 21:54:17.318545 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:17.318512 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" event={"ID":"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe","Type":"ContainerStarted","Data":"b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd"} Oct 30 21:54:17.318864 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:17.318632 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:17.910770 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:54:17.910732 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:54:21.291498 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:21.291455 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fj82z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:54:21.291874 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:21.291518 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:22.296233 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:22.296209 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-9zdg4" Oct 30 21:54:22.437398 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:22.437354 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" podStartSLOduration=6.961816544 podStartE2EDuration="12.437312746s" totalImagesPullingTime="5.475496202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:17.440560182 +0000 UTC m=+477.739970065" watchObservedRunningTime="2026-10-30 21:54:22.437312746 +0000 UTC m=+482.736722625" Oct 30 21:54:23.351247 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:23.351215 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:23.372134 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:23.372104 4991 generic.go:356] "Generic (PLEG): container finished" podID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerID="240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c" exitCode=0 Oct 30 21:54:23.372260 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:23.372178 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" event={"ID":"ff7805e9-b57f-41cf-b098-6e6e28f2dc73","Type":"ContainerDied","Data":"240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c"} Oct 30 21:54:23.372479 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:23.372466 4991 scope.go:122] "RemoveContainer" containerID="240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c" Oct 30 21:54:23.373500 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:23.373476 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:23.373560 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:23.373539 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:23.373596 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:23.373571 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:24.315852 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.315820 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4nlcm"] Oct 30 21:54:24.329352 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.319478 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" Oct 30 21:54:24.371124 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.371091 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4nlcm"] Oct 30 21:54:24.379661 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.379633 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" event={"ID":"ff7805e9-b57f-41cf-b098-6e6e28f2dc73","Type":"ContainerStarted","Data":"4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e"} Oct 30 21:54:24.397706 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.397685 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b7wsj\"" Oct 30 21:54:24.461931 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.461891 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2dt\" (UniqueName: \"kubernetes.io/projected/990c2661-5652-460a-b630-cc204a5c89bf-kube-api-access-vq2dt\") pod \"authorino-6f6944bcdd-4nlcm\" (UID: \"990c2661-5652-460a-b630-cc204a5c89bf\") " pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" Oct 30 21:54:24.515857 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.515820 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-844dd57557-7jtjc"] Oct 30 21:54:24.543669 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.543634 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.563359 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.563334 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2dt\" (UniqueName: \"kubernetes.io/projected/990c2661-5652-460a-b630-cc204a5c89bf-kube-api-access-vq2dt\") pod \"authorino-6f6944bcdd-4nlcm\" (UID: \"990c2661-5652-460a-b630-cc204a5c89bf\") " pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" Oct 30 21:54:24.574960 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.574904 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-844dd57557-7jtjc"] Oct 30 21:54:24.646775 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.646740 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2dt\" (UniqueName: \"kubernetes.io/projected/990c2661-5652-460a-b630-cc204a5c89bf-kube-api-access-vq2dt\") pod \"authorino-6f6944bcdd-4nlcm\" (UID: \"990c2661-5652-460a-b630-cc204a5c89bf\") " pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" Oct 30 21:54:24.664086 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.664057 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/868fdf52-6fd4-4c94-871c-313badd2fb38-openshift-service-ca\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.664222 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.664126 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrqt\" (UniqueName: \"kubernetes.io/projected/868fdf52-6fd4-4c94-871c-313badd2fb38-kube-api-access-zzrqt\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.664222 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.664166 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/868fdf52-6fd4-4c94-871c-313badd2fb38-metrics-certs\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.664298 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.664278 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/868fdf52-6fd4-4c94-871c-313badd2fb38-maas-api-tls\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.668967 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.668949 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" Oct 30 21:54:24.714843 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.714814 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4nlcm"] Oct 30 21:54:24.765637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.765605 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrqt\" (UniqueName: \"kubernetes.io/projected/868fdf52-6fd4-4c94-871c-313badd2fb38-kube-api-access-zzrqt\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.765786 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.765688 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/868fdf52-6fd4-4c94-871c-313badd2fb38-metrics-certs\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.765786 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.765758 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/868fdf52-6fd4-4c94-871c-313badd2fb38-maas-api-tls\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.765906 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.765830 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/868fdf52-6fd4-4c94-871c-313badd2fb38-openshift-service-ca\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.766653 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.766621 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/868fdf52-6fd4-4c94-871c-313badd2fb38-openshift-service-ca\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.768166 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.768141 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/868fdf52-6fd4-4c94-871c-313badd2fb38-maas-api-tls\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.768568 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.768546 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/868fdf52-6fd4-4c94-871c-313badd2fb38-metrics-certs\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.846723 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.846659 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrqt\" (UniqueName: \"kubernetes.io/projected/868fdf52-6fd4-4c94-871c-313badd2fb38-kube-api-access-zzrqt\") pod \"maas-api-844dd57557-7jtjc\" (UID: \"868fdf52-6fd4-4c94-871c-313badd2fb38\") " pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:24.858211 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:24.858189 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:25.081376 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.081342 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4nlcm"] Oct 30 21:54:25.088198 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:54:25.088172 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod990c2661_5652_460a_b630_cc204a5c89bf.slice/crio-f4a5c08f21a6a3528d42f317f365cff8579a104be9625140f550b5b870b34975 WatchSource:0}: Error finding container f4a5c08f21a6a3528d42f317f365cff8579a104be9625140f550b5b870b34975: Status 404 returned error can't find the container with id f4a5c08f21a6a3528d42f317f365cff8579a104be9625140f550b5b870b34975 Oct 30 21:54:25.200033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.199957 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79dd6cdc8f-pkg5m"] Oct 30 21:54:25.204805 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.204784 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.218985 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.218961 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:54:25.244342 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.241426 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fj82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:54:25.244342 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.241477 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 21:54:25.246045 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.245990 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-844dd57557-7jtjc"] Oct 30 21:54:25.248283 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:54:25.248263 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868fdf52_6fd4_4c94_871c_313badd2fb38.slice/crio-6ed96b3f28af94a532fb2fed9a71c1ffe8210095ede9da1de800d3146d5ca60b WatchSource:0}: Error finding container 6ed96b3f28af94a532fb2fed9a71c1ffe8210095ede9da1de800d3146d5ca60b: Status 404 returned error can't find the container with id 6ed96b3f28af94a532fb2fed9a71c1ffe8210095ede9da1de800d3146d5ca60b Oct 30 21:54:25.255833 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.255814 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 21:54:25.258355 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.258315 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79dd6cdc8f-pkg5m"] Oct 30 21:54:25.270395 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.270354 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0374cab-f2cc-47f8-8743-867605dc7cef-tls-cert\") pod \"authorino-79dd6cdc8f-pkg5m\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.270509 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.270465 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmzn\" (UniqueName: \"kubernetes.io/projected/a0374cab-f2cc-47f8-8743-867605dc7cef-kube-api-access-lbmzn\") pod \"authorino-79dd6cdc8f-pkg5m\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.371073 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.371042 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmzn\" (UniqueName: \"kubernetes.io/projected/a0374cab-f2cc-47f8-8743-867605dc7cef-kube-api-access-lbmzn\") pod \"authorino-79dd6cdc8f-pkg5m\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.371220 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.371103 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0374cab-f2cc-47f8-8743-867605dc7cef-tls-cert\") pod \"authorino-79dd6cdc8f-pkg5m\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.373424 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.373406 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0374cab-f2cc-47f8-8743-867605dc7cef-tls-cert\") pod \"authorino-79dd6cdc8f-pkg5m\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.385881 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.385834 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" event={"ID":"868fdf52-6fd4-4c94-871c-313badd2fb38","Type":"ContainerStarted","Data":"6ed96b3f28af94a532fb2fed9a71c1ffe8210095ede9da1de800d3146d5ca60b"} Oct 30 21:54:25.387003 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.386976 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" event={"ID":"990c2661-5652-460a-b630-cc204a5c89bf","Type":"ContainerStarted","Data":"f4a5c08f21a6a3528d42f317f365cff8579a104be9625140f550b5b870b34975"} Oct 30 21:54:25.442624 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.442597 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmzn\" (UniqueName: \"kubernetes.io/projected/a0374cab-f2cc-47f8-8743-867605dc7cef-kube-api-access-lbmzn\") pod \"authorino-79dd6cdc8f-pkg5m\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.517464 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.517435 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:54:25.912287 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:25.912209 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79dd6cdc8f-pkg5m"] Oct 30 21:54:25.918017 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:54:25.917992 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0374cab_f2cc_47f8_8743_867605dc7cef.slice/crio-2e9070ce1f0ae0761fb88bb922246c1cc318bdff8a123be43c82e09bf46f5cff WatchSource:0}: Error finding container 2e9070ce1f0ae0761fb88bb922246c1cc318bdff8a123be43c82e09bf46f5cff: Status 404 returned error can't find the container with id 2e9070ce1f0ae0761fb88bb922246c1cc318bdff8a123be43c82e09bf46f5cff Oct 30 21:54:26.219308 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:26.219232 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fj82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:54:26.219308 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:26.219286 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 21:54:26.393473 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:26.393449 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" event={"ID":"a0374cab-f2cc-47f8-8743-867605dc7cef","Type":"ContainerStarted","Data":"2e9070ce1f0ae0761fb88bb922246c1cc318bdff8a123be43c82e09bf46f5cff"} Oct 30 21:54:27.219167 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.219083 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fj82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:54:27.219167 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.219149 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 21:54:27.400530 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.400492 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" event={"ID":"868fdf52-6fd4-4c94-871c-313badd2fb38","Type":"ContainerStarted","Data":"c3c08f96b2fa84cdb371a5e1494192813e544b63f64e69d8504ebb6a7dd13bc6"} Oct 30 21:54:27.400985 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.400561 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:27.402237 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.402205 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" event={"ID":"990c2661-5652-460a-b630-cc204a5c89bf","Type":"ContainerStarted","Data":"0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f"} Oct 30 21:54:27.402390 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.402312 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" podUID="990c2661-5652-460a-b630-cc204a5c89bf" containerName="authorino" containerID="cri-o://0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f" gracePeriod=30 Oct 30 21:54:27.515343 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.515279 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" podStartSLOduration=2.443944386 podStartE2EDuration="3.515263526s" totalImagesPullingTime="1.07131914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:27.51206642 +0000 UTC m=+487.811476321" watchObservedRunningTime="2026-10-30 21:54:27.515263526 +0000 UTC m=+487.814673405" Oct 30 21:54:27.698935 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.698906 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" Oct 30 21:54:27.791650 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.791576 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990c2661-5652-460a-b630-cc204a5c89bf-kube-api-access-vq2dt\" (UniqueName: \"kubernetes.io/projected/990c2661-5652-460a-b630-cc204a5c89bf-kube-api-access-vq2dt\") pod \"990c2661-5652-460a-b630-cc204a5c89bf\" (UID: \"990c2661-5652-460a-b630-cc204a5c89bf\") " Oct 30 21:54:27.793687 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.793665 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990c2661-5652-460a-b630-cc204a5c89bf-kube-api-access-vq2dt" pod "990c2661-5652-460a-b630-cc204a5c89bf" (UID: "990c2661-5652-460a-b630-cc204a5c89bf"). InnerVolumeSpecName "kube-api-access-vq2dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:27.893025 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:27.893000 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq2dt\" (UniqueName: \"kubernetes.io/projected/990c2661-5652-460a-b630-cc204a5c89bf-kube-api-access-vq2dt\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.219193 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.219108 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fj82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:54:28.219371 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.219213 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 21:54:28.409733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.409698 4991 generic.go:356] "Generic (PLEG): container finished" podID="990c2661-5652-460a-b630-cc204a5c89bf" containerID="0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f" exitCode=0 Oct 30 21:54:28.410210 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.409761 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" Oct 30 21:54:28.410210 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.409783 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" event={"ID":"990c2661-5652-460a-b630-cc204a5c89bf","Type":"ContainerDied","Data":"0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f"} Oct 30 21:54:28.410210 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.409829 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4nlcm" event={"ID":"990c2661-5652-460a-b630-cc204a5c89bf","Type":"ContainerDied","Data":"f4a5c08f21a6a3528d42f317f365cff8579a104be9625140f550b5b870b34975"} Oct 30 21:54:28.410210 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.409854 4991 scope.go:122] "RemoveContainer" containerID="0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f" Oct 30 21:54:28.412629 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.412607 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" event={"ID":"a0374cab-f2cc-47f8-8743-867605dc7cef","Type":"ContainerStarted","Data":"7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812"} Oct 30 21:54:28.445507 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.445472 4991 scope.go:122] "RemoveContainer" containerID="0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f" Oct 30 21:54:28.445792 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:54:28.445773 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f\": container with ID starting with 0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f not found: ID does not exist" containerID="0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f" Oct 30 21:54:28.445877 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.445803 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f"} err="failed to get container status \"0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f\": rpc error: code = NotFound desc = could not find container \"0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f\": container with ID starting with 0e51410de72052681bb1fd0c583078941bbfab80134125c5a3ba86697866276f not found: ID does not exist" Oct 30 21:54:28.578555 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.578515 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4nlcm"] Oct 30 21:54:28.584133 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.584105 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4nlcm"] Oct 30 21:54:28.697032 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:28.696970 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" podStartSLOduration=2.153715293 podStartE2EDuration="3.696955806s" totalImagesPullingTime="1.543240513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:28.694739155 +0000 UTC m=+488.994149046" watchObservedRunningTime="2026-10-30 21:54:28.696955806 +0000 UTC m=+488.996365686" Oct 30 21:54:29.218810 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:29.218772 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fj82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:54:29.218980 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:29.218832 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 21:54:29.873136 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:29.873040 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990c2661-5652-460a-b630-cc204a5c89bf" path="/var/lib/kubelet/pods/990c2661-5652-460a-b630-cc204a5c89bf/volumes" Oct 30 21:54:30.223251 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:30.223176 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:54:30.223534 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:30.223515 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:54:30.224066 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:30.224048 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 21:54:33.417304 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.417274 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-844dd57557-7jtjc" Oct 30 21:54:33.590198 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.590154 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr"] Oct 30 21:54:33.590414 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.590386 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" podUID="c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" containerName="maas-api" containerID="cri-o://b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd" gracePeriod=30 Oct 30 21:54:33.839487 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.839459 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:33.944266 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.944238 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-maas-api-tls\") pod \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " Oct 30 21:54:33.944403 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.944292 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-metrics-certs\") pod \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " Oct 30 21:54:33.944403 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.944371 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-openshift-service-ca\") pod \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " Oct 30 21:54:33.944491 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.944424 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-kube-api-access-wtgc6\" (UniqueName: \"kubernetes.io/projected/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-kube-api-access-wtgc6\") pod \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\" (UID: \"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe\") " Oct 30 21:54:33.944771 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.944745 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-openshift-service-ca" pod "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" (UID: "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:33.946481 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.946443 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-metrics-certs" pod "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" (UID: "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:33.946570 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.946495 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-maas-api-tls" pod "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" (UID: "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:33.946570 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:33.946544 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-kube-api-access-wtgc6" pod "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" (UID: "c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe"). InnerVolumeSpecName "kube-api-access-wtgc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:34.045030 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.045001 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtgc6\" (UniqueName: \"kubernetes.io/projected/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-kube-api-access-wtgc6\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:34.045030 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.045028 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:34.045185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.045038 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:34.045185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.045048 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:34.445591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.445510 4991 generic.go:356] "Generic (PLEG): container finished" podID="c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" containerID="b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd" exitCode=0 Oct 30 21:54:34.445591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.445556 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" Oct 30 21:54:34.445591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.445563 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" event={"ID":"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe","Type":"ContainerDied","Data":"b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd"} Oct 30 21:54:34.446098 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.445611 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr" event={"ID":"c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe","Type":"ContainerDied","Data":"55c65d54bf01827ce6039cb95df425c70a2263d6c7b3fa6f576e676bee0bfc49"} Oct 30 21:54:34.446098 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.445635 4991 scope.go:122] "RemoveContainer" containerID="b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd" Oct 30 21:54:34.455065 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.455050 4991 scope.go:122] "RemoveContainer" containerID="b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd" Oct 30 21:54:34.455361 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:54:34.455345 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd\": container with ID starting with b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd not found: ID does not exist" containerID="b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd" Oct 30 21:54:34.455398 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.455371 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd"} err="failed to get container status \"b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd\": rpc error: code = NotFound desc = could not find container \"b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd\": container with ID starting with b813c0f82fbfd911723f4c049be3d83248bbd277b557e1cbf37e3771037148cd not found: ID does not exist" Oct 30 21:54:34.600937 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.600909 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr"] Oct 30 21:54:34.647927 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:34.647901 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-86xvr"] Oct 30 21:54:35.873848 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:54:35.873814 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" path="/var/lib/kubelet/pods/c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe/volumes" Oct 30 21:54:47.909382 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:54:47.909344 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:55:17.910078 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:55:17.910038 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:55:47.909636 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:55:47.909600 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:56:17.909625 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:56:17.909586 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:56:19.965260 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:19.965229 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:56:19.967590 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:19.967569 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 21:56:47.910651 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:56:47.910613 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:56:51.094473 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.094434 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8"] Oct 30 21:56:51.094913 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.094847 4991 cpu_manager.go:414] "containerMap: removing container" podUID="990c2661-5652-460a-b630-cc204a5c89bf" containerName="authorino" Oct 30 21:56:51.094913 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.094860 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990c2661-5652-460a-b630-cc204a5c89bf" containerName="authorino" Oct 30 21:56:51.094913 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.094875 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" containerName="maas-api" Oct 30 21:56:51.094913 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.094882 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" containerName="maas-api" Oct 30 21:56:51.095046 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.094983 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="990c2661-5652-460a-b630-cc204a5c89bf" containerName="authorino" Oct 30 21:56:51.095046 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.094995 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c548e5b4-d7c1-4a9e-88cd-17f5f7e6fdfe" containerName="maas-api" Oct 30 21:56:51.098284 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.098265 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.147838 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.147815 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:56:51.194147 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.194120 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:56:51.194274 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.194161 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-phhhm\"" Oct 30 21:56:51.194274 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.194216 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 21:56:51.196413 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.196395 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8"] Oct 30 21:56:51.214419 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.214394 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.214528 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.214452 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.214574 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.214524 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.214574 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.214543 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.214667 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.214587 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmcm\" (UniqueName: \"kubernetes.io/projected/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-kube-api-access-6xmcm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.214727 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.214701 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.315773 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.315740 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmcm\" (UniqueName: \"kubernetes.io/projected/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-kube-api-access-6xmcm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.315773 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.315776 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.315990 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.315810 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.315990 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.315873 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.315990 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.315902 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.315990 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.315923 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.316262 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.316238 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.316353 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.316274 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.316397 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.316356 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.318055 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.318040 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.318340 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.318302 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.376675 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.376620 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmcm\" (UniqueName: \"kubernetes.io/projected/0e5b737c-58c1-4ef3-a674-62eb4eb1447e-kube-api-access-6xmcm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kcnm8\" (UID: \"0e5b737c-58c1-4ef3-a674-62eb4eb1447e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.411140 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.411114 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:56:51.784406 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.784372 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8"] Oct 30 21:56:51.789819 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:56:51.789791 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5b737c_58c1_4ef3_a674_62eb4eb1447e.slice/crio-82fbea5892e6e675d0140726b09a975b7e509f4ee3904fe8636963b0d7b3323c WatchSource:0}: Error finding container 82fbea5892e6e675d0140726b09a975b7e509f4ee3904fe8636963b0d7b3323c: Status 404 returned error can't find the container with id 82fbea5892e6e675d0140726b09a975b7e509f4ee3904fe8636963b0d7b3323c Oct 30 21:56:51.791016 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:51.790996 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:56:52.154949 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:52.154849 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" event={"ID":"0e5b737c-58c1-4ef3-a674-62eb4eb1447e","Type":"ContainerStarted","Data":"82fbea5892e6e675d0140726b09a975b7e509f4ee3904fe8636963b0d7b3323c"} Oct 30 21:56:54.169271 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:54.169227 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" event={"ID":"0e5b737c-58c1-4ef3-a674-62eb4eb1447e","Type":"ContainerStarted","Data":"f091f5f7a7682b486d48a5a363e1de0af28fc78e1b269832dd97b1beba662bac"} Oct 30 21:56:54.280908 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:54.280846 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" podStartSLOduration=1.7267067680000001 podStartE2EDuration="3.280825621s" totalImagesPullingTime="1.554118853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:54.279526055 +0000 UTC m=+634.578935937" watchObservedRunningTime="2026-10-30 21:56:54.280825621 +0000 UTC m=+634.580235504" Oct 30 21:56:54.413363 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:56:54.412241 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:57:00.810215 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.810172 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp"] Oct 30 21:57:00.813963 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.813935 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:00.899562 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.899525 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:00.899716 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.899653 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7jg\" (UniqueName: \"kubernetes.io/projected/a617761d-3aee-4bf5-84ef-c98a111da859-kube-api-access-pd7jg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:00.899716 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.899691 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a617761d-3aee-4bf5-84ef-c98a111da859-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:00.899716 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.899708 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:00.899832 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.899805 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:00.899925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.899903 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:00.910253 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.910229 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 21:57:00.912523 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:00.912492 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp"] Oct 30 21:57:01.001345 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001279 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.001519 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001401 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7jg\" (UniqueName: \"kubernetes.io/projected/a617761d-3aee-4bf5-84ef-c98a111da859-kube-api-access-pd7jg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.001519 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001442 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a617761d-3aee-4bf5-84ef-c98a111da859-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.001519 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001466 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.001519 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001509 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.001744 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001571 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.001823 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001804 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.001954 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.001928 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.002259 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.002201 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.003898 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.003876 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617761d-3aee-4bf5-84ef-c98a111da859-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.004451 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.004433 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a617761d-3aee-4bf5-84ef-c98a111da859-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.063046 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.062982 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7jg\" (UniqueName: \"kubernetes.io/projected/a617761d-3aee-4bf5-84ef-c98a111da859-kube-api-access-pd7jg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp\" (UID: \"a617761d-3aee-4bf5-84ef-c98a111da859\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.127094 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.127063 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:01.411906 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.411494 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:57:01.429366 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.429340 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:57:01.501698 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:01.501660 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp"] Oct 30 21:57:01.508165 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:57:01.508132 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda617761d_3aee_4bf5_84ef_c98a111da859.slice/crio-ad570068a3e068a60c341696c36312d830702c1eb75cbca23f24beec10c6634d WatchSource:0}: Error finding container ad570068a3e068a60c341696c36312d830702c1eb75cbca23f24beec10c6634d: Status 404 returned error can't find the container with id ad570068a3e068a60c341696c36312d830702c1eb75cbca23f24beec10c6634d Oct 30 21:57:02.212703 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:02.212678 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" event={"ID":"a617761d-3aee-4bf5-84ef-c98a111da859","Type":"ContainerStarted","Data":"ad570068a3e068a60c341696c36312d830702c1eb75cbca23f24beec10c6634d"} Oct 30 21:57:02.234443 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:02.228056 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kcnm8" Oct 30 21:57:03.219577 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:03.219540 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" event={"ID":"a617761d-3aee-4bf5-84ef-c98a111da859","Type":"ContainerStarted","Data":"2d39d8e3112f2239f240081e7c61548ac6ad08d48d142866bb221de32a6bad11"} Oct 30 21:57:03.333525 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:03.333474 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" podStartSLOduration=2.692602437 podStartE2EDuration="3.333460632s" totalImagesPullingTime="640.858195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:03.33112218 +0000 UTC m=+643.630532065" watchObservedRunningTime="2026-10-30 21:57:03.333460632 +0000 UTC m=+643.632870511" Oct 30 21:57:04.127984 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:04.127941 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:07.825284 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.825236 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj"] Oct 30 21:57:07.831952 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.831932 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:07.928476 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.928432 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:57:07.930745 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.930719 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj"] Oct 30 21:57:07.973411 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.973388 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:07.973525 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.973439 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1175e1de-ccd9-486e-904e-ca0d060232b7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:07.973599 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.973550 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lnsj\" (UniqueName: \"kubernetes.io/projected/1175e1de-ccd9-486e-904e-ca0d060232b7-kube-api-access-5lnsj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:07.973599 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.973590 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:07.973703 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.973686 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:07.973746 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:07.973714 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074284 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074253 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074284 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074287 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074303 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074397 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1175e1de-ccd9-486e-904e-ca0d060232b7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074469 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lnsj\" (UniqueName: \"kubernetes.io/projected/1175e1de-ccd9-486e-904e-ca0d060232b7-kube-api-access-5lnsj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074524 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074497 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074757 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074731 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074819 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074769 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.074909 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.074840 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.076689 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.076637 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1175e1de-ccd9-486e-904e-ca0d060232b7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.076933 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.076914 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1175e1de-ccd9-486e-904e-ca0d060232b7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.137245 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.137178 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lnsj\" (UniqueName: \"kubernetes.io/projected/1175e1de-ccd9-486e-904e-ca0d060232b7-kube-api-access-5lnsj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj\" (UID: \"1175e1de-ccd9-486e-904e-ca0d060232b7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.146220 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.146190 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:08.545136 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:08.545100 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj"] Oct 30 21:57:08.550691 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:57:08.550656 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1175e1de_ccd9_486e_904e_ca0d060232b7.slice/crio-cfc8c2f95832b5124dc2c035bedec1b267b08c46a95b3ec941e72a2900ea0a0b WatchSource:0}: Error finding container cfc8c2f95832b5124dc2c035bedec1b267b08c46a95b3ec941e72a2900ea0a0b: Status 404 returned error can't find the container with id cfc8c2f95832b5124dc2c035bedec1b267b08c46a95b3ec941e72a2900ea0a0b Oct 30 21:57:09.255272 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:09.255242 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" event={"ID":"1175e1de-ccd9-486e-904e-ca0d060232b7","Type":"ContainerStarted","Data":"359f35ac95ab2e848334e34313ce0a86a2210aada6e1edb06e368626596c9b2b"} Oct 30 21:57:09.255678 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:09.255368 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" event={"ID":"1175e1de-ccd9-486e-904e-ca0d060232b7","Type":"ContainerStarted","Data":"cfc8c2f95832b5124dc2c035bedec1b267b08c46a95b3ec941e72a2900ea0a0b"} Oct 30 21:57:09.367720 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:09.367655 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" podStartSLOduration=1.756729022 podStartE2EDuration="2.367635108s" totalImagesPullingTime="610.906086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:09.366624677 +0000 UTC m=+649.666034557" watchObservedRunningTime="2026-10-30 21:57:09.367635108 +0000 UTC m=+649.667044991" Oct 30 21:57:10.146372 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:10.146340 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:11.127251 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.127217 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:11.139529 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.139506 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:11.277050 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.277021 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp" Oct 30 21:57:11.299556 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.299528 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t"] Oct 30 21:57:11.303299 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.303273 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.356024 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.355995 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:57:11.357124 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.357100 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t"] Oct 30 21:57:11.411750 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.411661 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.411750 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.411706 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.411987 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.411824 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhcx\" (UniqueName: \"kubernetes.io/projected/227fcbbf-d027-4db6-b8ec-03659183f431-kube-api-access-4bhcx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.411987 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.411962 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227fcbbf-d027-4db6-b8ec-03659183f431-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.412146 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.412017 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.412209 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.412186 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513020 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.512986 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhcx\" (UniqueName: \"kubernetes.io/projected/227fcbbf-d027-4db6-b8ec-03659183f431-kube-api-access-4bhcx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513062 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227fcbbf-d027-4db6-b8ec-03659183f431-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513099 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513131 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513312 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513208 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513312 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513231 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513582 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513564 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513799 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513562 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.513861 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.513796 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.515970 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.515947 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227fcbbf-d027-4db6-b8ec-03659183f431-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.516175 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.516157 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227fcbbf-d027-4db6-b8ec-03659183f431-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.571033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.571000 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhcx\" (UniqueName: \"kubernetes.io/projected/227fcbbf-d027-4db6-b8ec-03659183f431-kube-api-access-4bhcx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t\" (UID: \"227fcbbf-d027-4db6-b8ec-03659183f431\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.616843 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.616813 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:11.996042 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:11.996015 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t"] Oct 30 21:57:12.001765 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:57:12.001740 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod227fcbbf_d027_4db6_b8ec_03659183f431.slice/crio-6a4832c64b0fd4e8599ed38afcede937f2fe090216d540891c7519d364bb6b4d WatchSource:0}: Error finding container 6a4832c64b0fd4e8599ed38afcede937f2fe090216d540891c7519d364bb6b4d: Status 404 returned error can't find the container with id 6a4832c64b0fd4e8599ed38afcede937f2fe090216d540891c7519d364bb6b4d Oct 30 21:57:12.273191 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:12.273110 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" event={"ID":"227fcbbf-d027-4db6-b8ec-03659183f431","Type":"ContainerStarted","Data":"6a4832c64b0fd4e8599ed38afcede937f2fe090216d540891c7519d364bb6b4d"} Oct 30 21:57:13.281480 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:13.281441 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" event={"ID":"227fcbbf-d027-4db6-b8ec-03659183f431","Type":"ContainerStarted","Data":"86c61a7e27b62ad724db807e125727e8de342e645945623c9ab80eb3076fd031"} Oct 30 21:57:13.393399 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:13.393348 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podStartSLOduration=1.814815777 podStartE2EDuration="2.393334878s" totalImagesPullingTime="578.519101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:13.391903373 +0000 UTC m=+653.691313272" watchObservedRunningTime="2026-10-30 21:57:13.393334878 +0000 UTC m=+653.692744748" Oct 30 21:57:13.617530 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:13.617450 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:17.910494 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:57:17.910453 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:57:18.147019 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:18.146992 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:18.159934 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:18.159906 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:18.321959 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:18.321934 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj" Oct 30 21:57:21.617151 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:21.617115 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:21.629892 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:21.629866 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:22.344433 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:22.344407 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 21:57:25.159523 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.159492 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq"] Oct 30 21:57:25.164585 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.164561 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.237730 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.237702 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.237833 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.237733 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.237833 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.237764 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.237925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.237906 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.237962 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.237940 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.238005 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.237966 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw676\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-kube-api-access-qw676\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.238094 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.238076 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.238157 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.238111 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.238198 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.238163 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.252948 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.252921 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq"] Oct 30 21:57:25.338977 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.338947 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339150 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.338982 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339150 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339110 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339269 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339177 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339360 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339297 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339425 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339358 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw676\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-kube-api-access-qw676\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339425 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339412 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339551 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339429 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339551 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339443 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339551 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339494 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339721 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339700 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339830 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339804 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339890 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339849 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.339943 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.339895 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.341415 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.341390 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.341858 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.341838 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.401236 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.401198 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.401445 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.401426 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw676\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-kube-api-access-qw676\") pod \"maas-default-gateway-openshift-default-6f54779779-c77bq\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.477449 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.477425 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:25.905927 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.905853 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq"] Oct 30 21:57:25.911035 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:57:25.911010 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1857b8_ed6c_4398_a4a0_fe8bf04e2bf1.slice/crio-62339a82bfbb670b1f64f754d597a84ffdf2fed0616336434d9e70f3cc195606 WatchSource:0}: Error finding container 62339a82bfbb670b1f64f754d597a84ffdf2fed0616336434d9e70f3cc195606: Status 404 returned error can't find the container with id 62339a82bfbb670b1f64f754d597a84ffdf2fed0616336434d9e70f3cc195606 Oct 30 21:57:25.913070 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.913041 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:25.913169 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.913114 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:25.913230 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:25.913183 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:26.364797 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:26.364758 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" event={"ID":"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1","Type":"ContainerStarted","Data":"475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b"} Oct 30 21:57:26.365185 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:26.364823 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" event={"ID":"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1","Type":"ContainerStarted","Data":"62339a82bfbb670b1f64f754d597a84ffdf2fed0616336434d9e70f3cc195606"} Oct 30 21:57:26.477528 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:26.477494 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:26.479173 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:26.479145 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:26.479287 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:26.479191 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:57:26.481062 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:26.481022 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podStartSLOduration=1.48100911 podStartE2EDuration="1.48100911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:26.480221862 +0000 UTC m=+666.779631760" watchObservedRunningTime="2026-10-30 21:57:26.48100911 +0000 UTC m=+666.780418991" Oct 30 21:57:27.478661 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:27.478632 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:27.479056 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:27.478681 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:57:28.478491 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:28.478452 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:28.478660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:28.478506 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:57:29.478024 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:29.477988 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:29.478450 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:29.478047 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:57:30.478619 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:30.478582 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:30.479033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:30.478645 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:57:31.478474 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:31.478435 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:31.478675 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:31.478505 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:57:32.478219 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:32.478186 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:32.478408 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:32.478237 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:57:34.027801 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:34.027773 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:34.028356 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:34.028307 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:34.028877 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:34.028859 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 21:57:38.826946 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:38.826915 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57dbf74b78-dx9mj"] Oct 30 21:57:38.830486 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:38.830468 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:38.884527 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:38.884495 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57dbf74b78-dx9mj"] Oct 30 21:57:38.959671 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:38.959644 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd-tls-cert\") pod \"authorino-57dbf74b78-dx9mj\" (UID: \"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd\") " pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:38.959817 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:38.959702 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6b8m\" (UniqueName: \"kubernetes.io/projected/c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd-kube-api-access-c6b8m\") pod \"authorino-57dbf74b78-dx9mj\" (UID: \"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd\") " pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:39.060706 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:39.060678 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6b8m\" (UniqueName: \"kubernetes.io/projected/c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd-kube-api-access-c6b8m\") pod \"authorino-57dbf74b78-dx9mj\" (UID: \"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd\") " pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:39.060841 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:39.060746 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd-tls-cert\") pod \"authorino-57dbf74b78-dx9mj\" (UID: \"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd\") " pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:39.063095 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:39.063073 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd-tls-cert\") pod \"authorino-57dbf74b78-dx9mj\" (UID: \"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd\") " pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:39.119971 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:39.119910 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6b8m\" (UniqueName: \"kubernetes.io/projected/c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd-kube-api-access-c6b8m\") pod \"authorino-57dbf74b78-dx9mj\" (UID: \"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd\") " pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:39.142837 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:39.142815 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57dbf74b78-dx9mj" Oct 30 21:57:39.518656 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:39.518622 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57dbf74b78-dx9mj"] Oct 30 21:57:39.524363 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:57:39.524331 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6e9aa35_1c5f_4ff2_a5b7_eac59cf341bd.slice/crio-85d79833a3db78c80ba3acb4b95272443b27bfb78a870d7ae1fc3acf065eba27 WatchSource:0}: Error finding container 85d79833a3db78c80ba3acb4b95272443b27bfb78a870d7ae1fc3acf065eba27: Status 404 returned error can't find the container with id 85d79833a3db78c80ba3acb4b95272443b27bfb78a870d7ae1fc3acf065eba27 Oct 30 21:57:40.443740 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:40.443705 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57dbf74b78-dx9mj" event={"ID":"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd","Type":"ContainerStarted","Data":"85d79833a3db78c80ba3acb4b95272443b27bfb78a870d7ae1fc3acf065eba27"} Oct 30 21:57:41.453447 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.453414 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57dbf74b78-dx9mj" event={"ID":"c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd","Type":"ContainerStarted","Data":"1a754411dd162bc0188f39c17b7c52281850020bdae25751b3ec495b50f30d97"} Oct 30 21:57:41.562776 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.562686 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57dbf74b78-dx9mj" podStartSLOduration=1.831570455 podStartE2EDuration="3.562671572s" totalImagesPullingTime="1.731101117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:41.561125115 +0000 UTC m=+681.860534995" watchObservedRunningTime="2026-10-30 21:57:41.562671572 +0000 UTC m=+681.862081451" Oct 30 21:57:41.610832 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.610801 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79dd6cdc8f-pkg5m"] Oct 30 21:57:41.610985 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.610964 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" podUID="a0374cab-f2cc-47f8-8743-867605dc7cef" containerName="authorino" containerID="cri-o://7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812" gracePeriod=30 Oct 30 21:57:41.860566 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.860543 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:57:41.990330 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.990301 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0374cab-f2cc-47f8-8743-867605dc7cef-tls-cert\" (UniqueName: \"kubernetes.io/secret/a0374cab-f2cc-47f8-8743-867605dc7cef-tls-cert\") pod \"a0374cab-f2cc-47f8-8743-867605dc7cef\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " Oct 30 21:57:41.990497 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.990365 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0374cab-f2cc-47f8-8743-867605dc7cef-kube-api-access-lbmzn\" (UniqueName: \"kubernetes.io/projected/a0374cab-f2cc-47f8-8743-867605dc7cef-kube-api-access-lbmzn\") pod \"a0374cab-f2cc-47f8-8743-867605dc7cef\" (UID: \"a0374cab-f2cc-47f8-8743-867605dc7cef\") " Oct 30 21:57:41.992972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:41.992928 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0374cab-f2cc-47f8-8743-867605dc7cef-kube-api-access-lbmzn" pod "a0374cab-f2cc-47f8-8743-867605dc7cef" (UID: "a0374cab-f2cc-47f8-8743-867605dc7cef"). InnerVolumeSpecName "kube-api-access-lbmzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:42.002717 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.002688 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0374cab-f2cc-47f8-8743-867605dc7cef-tls-cert" pod "a0374cab-f2cc-47f8-8743-867605dc7cef" (UID: "a0374cab-f2cc-47f8-8743-867605dc7cef"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:42.091719 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.091638 4991 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0374cab-f2cc-47f8-8743-867605dc7cef-tls-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:42.091719 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.091675 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmzn\" (UniqueName: \"kubernetes.io/projected/a0374cab-f2cc-47f8-8743-867605dc7cef-kube-api-access-lbmzn\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:42.460861 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.460769 4991 generic.go:356] "Generic (PLEG): container finished" podID="a0374cab-f2cc-47f8-8743-867605dc7cef" containerID="7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812" exitCode=0 Oct 30 21:57:42.460861 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.460821 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" Oct 30 21:57:42.460861 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.460848 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" event={"ID":"a0374cab-f2cc-47f8-8743-867605dc7cef","Type":"ContainerDied","Data":"7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812"} Oct 30 21:57:42.461567 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.460890 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79dd6cdc8f-pkg5m" event={"ID":"a0374cab-f2cc-47f8-8743-867605dc7cef","Type":"ContainerDied","Data":"2e9070ce1f0ae0761fb88bb922246c1cc318bdff8a123be43c82e09bf46f5cff"} Oct 30 21:57:42.461567 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.460906 4991 scope.go:122] "RemoveContainer" containerID="7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812" Oct 30 21:57:42.470510 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.470485 4991 scope.go:122] "RemoveContainer" containerID="7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812" Oct 30 21:57:42.470798 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:57:42.470779 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812\": container with ID starting with 7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812 not found: ID does not exist" containerID="7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812" Oct 30 21:57:42.470866 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.470807 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812"} err="failed to get container status \"7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812\": rpc error: code = NotFound desc = could not find container \"7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812\": container with ID starting with 7d9f84d42b3d5212306ca77c811f6d2c0074d8de7879489829f78feb60b33812 not found: ID does not exist" Oct 30 21:57:42.625715 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.625681 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79dd6cdc8f-pkg5m"] Oct 30 21:57:42.673660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:42.673625 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79dd6cdc8f-pkg5m"] Oct 30 21:57:43.874340 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:57:43.874280 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0374cab-f2cc-47f8-8743-867605dc7cef" path="/var/lib/kubelet/pods/a0374cab-f2cc-47f8-8743-867605dc7cef/volumes" Oct 30 21:57:47.909759 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:57:47.909725 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:58:15.576669 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.576632 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z"] Oct 30 21:58:15.577083 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.577028 4991 cpu_manager.go:414] "containerMap: removing container" podUID="a0374cab-f2cc-47f8-8743-867605dc7cef" containerName="authorino" Oct 30 21:58:15.577083 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.577040 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0374cab-f2cc-47f8-8743-867605dc7cef" containerName="authorino" Oct 30 21:58:15.577154 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.577125 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0374cab-f2cc-47f8-8743-867605dc7cef" containerName="authorino" Oct 30 21:58:15.581765 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.581740 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.664860 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.664826 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z"] Oct 30 21:58:15.696572 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696538 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-certs\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.696709 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696640 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-envoy\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.696709 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696671 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-credential-socket\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.696791 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696763 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istiod-ca-cert\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.696850 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696799 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-data\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.696917 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696892 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-socket\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.696978 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696943 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7ms\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-kube-api-access-7g7ms\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.697016 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.696984 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-token\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.697068 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.697021 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-podinfo\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.706248 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.706228 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2a41d4db-openshift-default-dockercfg-5fd67\"" Oct 30 21:58:15.798506 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798482 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-envoy\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798512 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-credential-socket\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798538 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istiod-ca-cert\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798559 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-data\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798579 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-socket\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798599 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7ms\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-kube-api-access-7g7ms\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798625 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-token\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798955 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798662 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-podinfo\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.798955 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798726 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-certs\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.799065 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.798994 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-data\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.799065 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.799047 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-credential-socket\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.799171 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.799130 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-socket\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.799277 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.799258 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-certs\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.799439 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.799420 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istiod-ca-cert\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.800738 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.800715 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-envoy\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.801131 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.801109 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-podinfo\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.859940 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.859885 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-token\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.860352 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.860334 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7ms\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-kube-api-access-7g7ms\") pod \"e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:15.898108 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:15.898055 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:16.250159 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.250128 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9"] Oct 30 21:58:16.255031 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.255010 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303396 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303365 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-data\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303573 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303428 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849162f4-291a-445d-965d-63579ecc637b-istiod-ca-cert\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303573 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303536 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txb6k\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-kube-api-access-txb6k\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303699 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303576 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-certs\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303699 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303605 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849162f4-291a-445d-965d-63579ecc637b-istio-podinfo\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303699 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303653 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-credential-socket\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303699 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303684 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-socket\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303898 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303708 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-istio-token\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.303898 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.303741 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-envoy\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.304354 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.304315 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4b2e37ae-gw-openshift-default-dockercfg-xsqkp\"" Oct 30 21:58:16.312054 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.312028 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9"] Oct 30 21:58:16.391150 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.391124 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z"] Oct 30 21:58:16.398048 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:16.398019 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbd8d30_28e1_44d4_880c_25bcd3e67696.slice/crio-dbc1ce71586dc6813c9e49a794096d4da40d16fcd3a5d5ba19e865f1aaaee97d WatchSource:0}: Error finding container dbc1ce71586dc6813c9e49a794096d4da40d16fcd3a5d5ba19e865f1aaaee97d: Status 404 returned error can't find the container with id dbc1ce71586dc6813c9e49a794096d4da40d16fcd3a5d5ba19e865f1aaaee97d Oct 30 21:58:16.402005 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.401969 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:16.402198 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.402173 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:16.403078 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.403050 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:16.405522 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405499 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-credential-socket\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.405660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405536 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-socket\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.405660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405565 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-istio-token\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.405660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405597 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-envoy\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.406014 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405690 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-data\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.406014 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405753 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849162f4-291a-445d-965d-63579ecc637b-istiod-ca-cert\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.406014 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405785 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txb6k\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-kube-api-access-txb6k\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.406014 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405816 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-certs\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.406014 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.405839 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849162f4-291a-445d-965d-63579ecc637b-istio-podinfo\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.406840 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.406813 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-data\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.407388 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.407081 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-socket\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.407388 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.407348 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-credential-socket\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.407922 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.407853 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-certs\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.408581 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.408558 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849162f4-291a-445d-965d-63579ecc637b-istiod-ca-cert\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.410061 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.410038 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-envoy\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.410990 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.410940 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849162f4-291a-445d-965d-63579ecc637b-istio-podinfo\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.464903 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.464882 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-istio-token\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.465909 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.465886 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txb6k\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-kube-api-access-txb6k\") pod \"e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.569098 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.569015 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:16.674737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.674696 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" event={"ID":"4fbd8d30-28e1-44d4-880c-25bcd3e67696","Type":"ContainerStarted","Data":"2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a"} Oct 30 21:58:16.675175 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.674773 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" event={"ID":"4fbd8d30-28e1-44d4-880c-25bcd3e67696","Type":"ContainerStarted","Data":"dbc1ce71586dc6813c9e49a794096d4da40d16fcd3a5d5ba19e865f1aaaee97d"} Oct 30 21:58:16.790978 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.790915 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podStartSLOduration=1.790898141 podStartE2EDuration="1.790898141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:16.788809916 +0000 UTC m=+717.088219821" watchObservedRunningTime="2026-10-30 21:58:16.790898141 +0000 UTC m=+717.090308022" Oct 30 21:58:16.898203 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.898137 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:16.995717 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:16.995687 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9"] Oct 30 21:58:17.001132 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:17.001104 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849162f4_291a_445d_965d_63579ecc637b.slice/crio-748307d1f9598a15da94dcd46c715c69e45b7aaca82ff03137c298856e0e2feb WatchSource:0}: Error finding container 748307d1f9598a15da94dcd46c715c69e45b7aaca82ff03137c298856e0e2feb: Status 404 returned error can't find the container with id 748307d1f9598a15da94dcd46c715c69e45b7aaca82ff03137c298856e0e2feb Oct 30 21:58:17.003350 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:17.003304 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:17.003457 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:17.003381 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:17.003457 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:17.003409 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:17.687912 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:17.687868 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" event={"ID":"849162f4-291a-445d-965d-63579ecc637b","Type":"ContainerStarted","Data":"c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4"} Oct 30 21:58:17.688392 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:17.687965 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" event={"ID":"849162f4-291a-445d-965d-63579ecc637b","Type":"ContainerStarted","Data":"748307d1f9598a15da94dcd46c715c69e45b7aaca82ff03137c298856e0e2feb"} Oct 30 21:58:17.805548 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:17.805494 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podStartSLOduration=1.805479739 podStartE2EDuration="1.805479739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:17.802511223 +0000 UTC m=+718.101921120" watchObservedRunningTime="2026-10-30 21:58:17.805479739 +0000 UTC m=+718.104889618" Oct 30 21:58:17.904994 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:17.904960 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:17.916362 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:17.916306 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:58:18.569854 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:18.569817 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:18.574917 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:18.574893 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:18.693996 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:18.693961 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:18.694422 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:18.694017 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:18.694631 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:18.694608 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:18.694712 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:18.694664 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:58:23.143488 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.143455 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q"] Oct 30 21:58:23.148854 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.148834 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.229273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.229240 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q"] Oct 30 21:58:23.277836 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.277804 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-podinfo\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278015 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.277841 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-envoy\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278015 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.277968 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvc9x\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-kube-api-access-hvc9x\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278015 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.277994 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-credential-socket\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278152 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.278073 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-data\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278152 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.278125 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-certs\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278229 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.278178 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf81202-ac7f-478e-9819-2e3030e9bd78-istiod-ca-cert\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278229 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.278195 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-token\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.278229 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.278218 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-socket\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.286839 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.286815 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4705e99c-openshift-default-dockercfg-2s8x5\"" Oct 30 21:58:23.379653 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379609 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-podinfo\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.379653 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379658 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-envoy\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.379925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379722 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvc9x\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-kube-api-access-hvc9x\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.379925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379741 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-credential-socket\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.379925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379789 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-data\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.379925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379840 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-certs\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.379925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379898 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf81202-ac7f-478e-9819-2e3030e9bd78-istiod-ca-cert\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.379925 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379925 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-token\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.380244 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.379963 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-socket\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.380304 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.380231 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-credential-socket\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.380380 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.380298 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-data\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.380481 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.380459 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-certs\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.380565 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.380542 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-socket\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.380759 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.380741 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf81202-ac7f-478e-9819-2e3030e9bd78-istiod-ca-cert\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.382590 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.382567 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-envoy\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.382723 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.382706 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-podinfo\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.440171 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.440105 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-token\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.440305 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.440255 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvc9x\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-kube-api-access-hvc9x\") pod \"e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.461643 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.461615 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:23.888382 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.888352 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q"] Oct 30 21:58:23.893549 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:23.893522 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf81202_ac7f_478e_9819_2e3030e9bd78.slice/crio-75f4053a55b640014a80cb0707ede0ebcdd0c194c3502e41173f2ecfe69910ee WatchSource:0}: Error finding container 75f4053a55b640014a80cb0707ede0ebcdd0c194c3502e41173f2ecfe69910ee: Status 404 returned error can't find the container with id 75f4053a55b640014a80cb0707ede0ebcdd0c194c3502e41173f2ecfe69910ee Oct 30 21:58:23.895627 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.895599 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:23.895687 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.895656 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:23.895727 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:23.895690 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:24.738078 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.738038 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" event={"ID":"7bf81202-ac7f-478e-9819-2e3030e9bd78","Type":"ContainerStarted","Data":"4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c"} Oct 30 21:58:24.738669 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.738101 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" event={"ID":"7bf81202-ac7f-478e-9819-2e3030e9bd78","Type":"ContainerStarted","Data":"75f4053a55b640014a80cb0707ede0ebcdd0c194c3502e41173f2ecfe69910ee"} Oct 30 21:58:24.792890 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.792853 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx"] Oct 30 21:58:24.798643 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.798622 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.850022 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.849997 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx"] Oct 30 21:58:24.883814 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.883793 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-63d97fec-serving-cert\"" Oct 30 21:58:24.897257 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.897233 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-openshift-service-ca\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.897360 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.897305 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b987\" (UniqueName: \"kubernetes.io/projected/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-kube-api-access-8b987\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.897416 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.897385 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.897461 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.897442 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-metrics-certs\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.998891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.998811 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b987\" (UniqueName: \"kubernetes.io/projected/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-kube-api-access-8b987\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.998891 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.998862 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.999120 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.998900 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-metrics-certs\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.999120 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.998948 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-openshift-service-ca\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:24.999120 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:24.999048 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-serving-cert: secret "maas-api-e2e-ait-63d97fec-serving-cert" not found Oct 30 21:58:24.999120 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:24.999111 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls podName:f7b606d7-cfb7-4f93-80ed-63c5f77802d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:58:25.499095028 +0000 UTC m=+725.798504906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls") pod "maas-api-e2e-ait-63d97fec-6668946754-bggcx" (UID: "f7b606d7-cfb7-4f93-80ed-63c5f77802d5") : secret "maas-api-e2e-ait-63d97fec-serving-cert" not found Oct 30 21:58:24.999795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:24.999776 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-openshift-service-ca\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:25.001581 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.001555 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-metrics-certs\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:25.071124 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.071053 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podStartSLOduration=2.071031665 podStartE2EDuration="2.071031665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:24.93046439 +0000 UTC m=+725.229874287" watchObservedRunningTime="2026-10-30 21:58:25.071031665 +0000 UTC m=+725.370441546" Oct 30 21:58:25.074958 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.074928 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b987\" (UniqueName: \"kubernetes.io/projected/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-kube-api-access-8b987\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:25.462461 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.462367 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:25.467865 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.467842 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:25.505528 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.505496 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:25.507884 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.507858 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls\") pod \"maas-api-e2e-ait-63d97fec-6668946754-bggcx\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:25.712617 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.712529 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:25.746260 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.746233 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:25.747349 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:25.747300 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:58:26.088448 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:26.088416 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx"] Oct 30 21:58:26.094864 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:26.094839 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b606d7_cfb7_4f93_80ed_63c5f77802d5.slice/crio-b05c1704654f310fe03de8553e7d4dcec00da8d59c21a4257809ffc529fd637d WatchSource:0}: Error finding container b05c1704654f310fe03de8553e7d4dcec00da8d59c21a4257809ffc529fd637d: Status 404 returned error can't find the container with id b05c1704654f310fe03de8553e7d4dcec00da8d59c21a4257809ffc529fd637d Oct 30 21:58:26.752769 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:26.752737 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" event={"ID":"f7b606d7-cfb7-4f93-80ed-63c5f77802d5","Type":"ContainerStarted","Data":"b05c1704654f310fe03de8553e7d4dcec00da8d59c21a4257809ffc529fd637d"} Oct 30 21:58:27.760295 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:27.760256 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" event={"ID":"f7b606d7-cfb7-4f93-80ed-63c5f77802d5","Type":"ContainerStarted","Data":"b218afe023b7dd672e92285fc6b30ae3a5e366eb6381c8077c098dc8b9158f3d"} Oct 30 21:58:27.760751 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:27.760520 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:27.872879 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:27.872827 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" podStartSLOduration=2.96648565 podStartE2EDuration="3.872813146s" totalImagesPullingTime="906.327496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:27.870898175 +0000 UTC m=+728.170308069" watchObservedRunningTime="2026-10-30 21:58:27.872813146 +0000 UTC m=+728.172223039" Oct 30 21:58:29.728035 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.727995 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh"] Oct 30 21:58:29.732093 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.732074 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.781256 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.781233 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0b0026e9-serving-cert\"" Oct 30 21:58:29.809657 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.809627 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh"] Oct 30 21:58:29.856096 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.856067 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-openshift-service-ca\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.856386 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.856343 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.856386 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.856388 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8t8\" (UniqueName: \"kubernetes.io/projected/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-kube-api-access-9n8t8\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.856585 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.856564 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-metrics-certs\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.957792 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.957759 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-metrics-certs\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.957948 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.957827 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-openshift-service-ca\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.957948 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.957898 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.957948 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.957925 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8t8\" (UniqueName: \"kubernetes.io/projected/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-kube-api-access-9n8t8\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.958053 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:29.958032 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-serving-cert: secret "maas-api-e2e-ait-0b0026e9-serving-cert" not found Oct 30 21:58:29.958112 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:29.958101 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls podName:3a40511a-c02b-4b40-8c9b-f4a2af3a784c nodeName:}" failed. No retries permitted until 2026-10-30 21:58:30.458085557 +0000 UTC m=+730.757495420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls") pod "maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" (UID: "3a40511a-c02b-4b40-8c9b-f4a2af3a784c") : secret "maas-api-e2e-ait-0b0026e9-serving-cert" not found Oct 30 21:58:29.958545 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.958519 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-openshift-service-ca\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:29.960172 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:29.960149 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-metrics-certs\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:30.017818 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:30.017753 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8t8\" (UniqueName: \"kubernetes.io/projected/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-kube-api-access-9n8t8\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:30.463228 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:30.463142 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:30.465505 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:30.465484 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls\") pod \"maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:30.644919 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:30.644885 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:31.021896 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:31.021861 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh"] Oct 30 21:58:31.027863 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:31.027833 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a40511a_c02b_4b40_8c9b_f4a2af3a784c.slice/crio-464c71de33b5c93ee14844ab29403af66d0df694d2bfcdca0b45f7562655e0b4 WatchSource:0}: Error finding container 464c71de33b5c93ee14844ab29403af66d0df694d2bfcdca0b45f7562655e0b4: Status 404 returned error can't find the container with id 464c71de33b5c93ee14844ab29403af66d0df694d2bfcdca0b45f7562655e0b4 Oct 30 21:58:31.783277 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:31.783237 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" event={"ID":"3a40511a-c02b-4b40-8c9b-f4a2af3a784c","Type":"ContainerStarted","Data":"464c71de33b5c93ee14844ab29403af66d0df694d2bfcdca0b45f7562655e0b4"} Oct 30 21:58:32.836824 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:32.836772 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" event={"ID":"3a40511a-c02b-4b40-8c9b-f4a2af3a784c","Type":"ContainerStarted","Data":"def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2"} Oct 30 21:58:32.837841 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:32.837813 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:32.967156 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:32.967090 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" podStartSLOduration=3.042558815 podStartE2EDuration="3.96707169s" totalImagesPullingTime="924.512875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:32.965758315 +0000 UTC m=+733.265168195" watchObservedRunningTime="2026-10-30 21:58:32.96707169 +0000 UTC m=+733.266481570" Oct 30 21:58:33.771679 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:33.771649 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:58:34.571103 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.571073 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54"] Oct 30 21:58:34.575022 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.575000 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.624423 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.624401 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4b2e37ae-serving-cert\"" Oct 30 21:58:34.673343 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.673298 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54"] Oct 30 21:58:34.714854 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.714828 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2vw\" (UniqueName: \"kubernetes.io/projected/83e6a169-ba2b-4a5b-b904-21b34d262b01-kube-api-access-hm2vw\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.714988 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.714881 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-metrics-certs\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.715062 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.715000 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.715116 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.715064 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6a169-ba2b-4a5b-b904-21b34d262b01-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.815576 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.815548 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.815712 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.815595 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6a169-ba2b-4a5b-b904-21b34d262b01-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.815712 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.815670 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2vw\" (UniqueName: \"kubernetes.io/projected/83e6a169-ba2b-4a5b-b904-21b34d262b01-kube-api-access-hm2vw\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.815712 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.815696 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-metrics-certs\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.815875 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:34.815710 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-serving-cert: secret "maas-api-e2e-migrate-4b2e37ae-serving-cert" not found Oct 30 21:58:34.815875 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:34.815784 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls podName:83e6a169-ba2b-4a5b-b904-21b34d262b01 nodeName:}" failed. No retries permitted until 2026-10-30 21:58:35.31576274 +0000 UTC m=+735.615172618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls") pod "maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" (UID: "83e6a169-ba2b-4a5b-b904-21b34d262b01") : secret "maas-api-e2e-migrate-4b2e37ae-serving-cert" not found Oct 30 21:58:34.816387 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.816366 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6a169-ba2b-4a5b-b904-21b34d262b01-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.818441 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.818417 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-metrics-certs\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:34.873791 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:34.873723 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2vw\" (UniqueName: \"kubernetes.io/projected/83e6a169-ba2b-4a5b-b904-21b34d262b01-kube-api-access-hm2vw\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:35.323072 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:35.323034 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:35.325547 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:35.325517 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls\") pod \"maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:35.488001 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:35.487975 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:35.859467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:35.859440 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54"] Oct 30 21:58:35.864925 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:35.864893 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e6a169_ba2b_4a5b_b904_21b34d262b01.slice/crio-07d051ebb135ac26c50586a603696550a95bb55f43a1eb0135faf911175d040f WatchSource:0}: Error finding container 07d051ebb135ac26c50586a603696550a95bb55f43a1eb0135faf911175d040f: Status 404 returned error can't find the container with id 07d051ebb135ac26c50586a603696550a95bb55f43a1eb0135faf911175d040f Oct 30 21:58:36.863555 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:36.863517 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" event={"ID":"83e6a169-ba2b-4a5b-b904-21b34d262b01","Type":"ContainerStarted","Data":"1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba"} Oct 30 21:58:36.863955 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:36.863761 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:36.863955 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:36.863819 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" event={"ID":"83e6a169-ba2b-4a5b-b904-21b34d262b01","Type":"ContainerStarted","Data":"07d051ebb135ac26c50586a603696550a95bb55f43a1eb0135faf911175d040f"} Oct 30 21:58:36.988572 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:36.988513 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" podStartSLOduration=2.089007857 podStartE2EDuration="2.988500117s" totalImagesPullingTime="899.49226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:36.98695105 +0000 UTC m=+737.286360930" watchObservedRunningTime="2026-10-30 21:58:36.988500117 +0000 UTC m=+737.287909996" Oct 30 21:58:38.981108 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:38.981073 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j"] Oct 30 21:58:38.985167 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:38.985146 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.036669 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.036638 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2a41d4db-serving-cert\"" Oct 30 21:58:39.056208 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.056179 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-metrics-certs\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.056352 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.056239 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.056352 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.056314 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff15627-966f-4f5b-85f3-272af51a24da-openshift-service-ca\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.056454 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.056417 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wvz\" (UniqueName: \"kubernetes.io/projected/4ff15627-966f-4f5b-85f3-272af51a24da-kube-api-access-n2wvz\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.085980 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.085956 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j"] Oct 30 21:58:39.157429 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.157395 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-metrics-certs\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.157595 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.157457 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.157595 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.157476 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff15627-966f-4f5b-85f3-272af51a24da-openshift-service-ca\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.157595 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:39.157556 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-serving-cert: secret "maas-api-e2e-ait-2a41d4db-serving-cert" not found Oct 30 21:58:39.157800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.157593 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wvz\" (UniqueName: \"kubernetes.io/projected/4ff15627-966f-4f5b-85f3-272af51a24da-kube-api-access-n2wvz\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.157800 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:39.157624 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls podName:4ff15627-966f-4f5b-85f3-272af51a24da nodeName:}" failed. No retries permitted until 2026-10-30 21:58:39.657605345 +0000 UTC m=+739.957015205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls") pod "maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" (UID: "4ff15627-966f-4f5b-85f3-272af51a24da") : secret "maas-api-e2e-ait-2a41d4db-serving-cert" not found Oct 30 21:58:39.158111 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.158092 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff15627-966f-4f5b-85f3-272af51a24da-openshift-service-ca\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.159954 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.159933 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-metrics-certs\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.215863 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.215831 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wvz\" (UniqueName: \"kubernetes.io/projected/4ff15627-966f-4f5b-85f3-272af51a24da-kube-api-access-n2wvz\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.663255 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.663219 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.665739 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.665709 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls\") pod \"maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:39.854150 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.854121 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:58:39.898481 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:39.898451 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:40.278844 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:40.278811 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j"] Oct 30 21:58:40.283799 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:40.283773 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff15627_966f_4f5b_85f3_272af51a24da.slice/crio-608dae5826244dad5ce3b4f83a1754a384aa9745a83d44971f17160d672e86b8 WatchSource:0}: Error finding container 608dae5826244dad5ce3b4f83a1754a384aa9745a83d44971f17160d672e86b8: Status 404 returned error can't find the container with id 608dae5826244dad5ce3b4f83a1754a384aa9745a83d44971f17160d672e86b8 Oct 30 21:58:40.886531 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:40.886500 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" event={"ID":"4ff15627-966f-4f5b-85f3-272af51a24da","Type":"ContainerStarted","Data":"608dae5826244dad5ce3b4f83a1754a384aa9745a83d44971f17160d672e86b8"} Oct 30 21:58:41.893912 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:41.893852 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" event={"ID":"4ff15627-966f-4f5b-85f3-272af51a24da","Type":"ContainerStarted","Data":"8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2"} Oct 30 21:58:41.894453 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:41.893929 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:42.015176 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:42.015125 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" podStartSLOduration=3.06972635 podStartE2EDuration="4.015111651s" totalImagesPullingTime="945.385301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:42.014096677 +0000 UTC m=+742.313506582" watchObservedRunningTime="2026-10-30 21:58:42.015111651 +0000 UTC m=+742.314521531" Oct 30 21:58:42.876700 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:42.876665 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:58:47.907991 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:47.907964 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:58:47.911242 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:47.911217 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:58:49.735481 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:49.735436 4991 patch_prober.go:29] interesting pod/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:58:49.736005 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:49.735515 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:58:50.974186 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:50.974155 4991 generic.go:356] "Generic (PLEG): container finished" podID="849162f4-291a-445d-965d-63579ecc637b" containerID="c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4" exitCode=0 Oct 30 21:58:50.974653 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:50.974210 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" event={"ID":"849162f4-291a-445d-965d-63579ecc637b","Type":"ContainerDied","Data":"c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4"} Oct 30 21:58:50.974653 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:50.974613 4991 scope.go:122] "RemoveContainer" containerID="c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4" Oct 30 21:58:50.975858 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:50.975830 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:50.975918 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:50.975903 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:50.975953 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:50.975933 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:58:51.981467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:51.981429 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" event={"ID":"849162f4-291a-445d-965d-63579ecc637b","Type":"ContainerStarted","Data":"366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a"} Oct 30 21:58:52.576366 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:52.576302 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:52.576591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:52.576470 4991 patch_prober.go:29] interesting pod/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:58:52.576591 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:52.576517 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:58:53.170362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.170296 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx"] Oct 30 21:58:53.175273 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.175251 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.261060 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.261021 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx"] Oct 30 21:58:53.289301 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.289183 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-metrics-certs\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.289680 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.289656 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4tbk\" (UniqueName: \"kubernetes.io/projected/519f6640-0db9-43ac-a221-fd8c2def0a81-kube-api-access-r4tbk\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.289900 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.289885 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.289998 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.289985 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519f6640-0db9-43ac-a221-fd8c2def0a81-openshift-service-ca\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.292911 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.292888 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4705e99c-serving-cert\"" Oct 30 21:58:53.391243 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.391206 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tbk\" (UniqueName: \"kubernetes.io/projected/519f6640-0db9-43ac-a221-fd8c2def0a81-kube-api-access-r4tbk\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.391675 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.391651 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.392473 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:53.391781 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-serving-cert: secret "maas-api-e2e-derive-4705e99c-serving-cert" not found Oct 30 21:58:53.392473 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:58:53.391884 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls podName:519f6640-0db9-43ac-a221-fd8c2def0a81 nodeName:}" failed. No retries permitted until 2026-10-30 21:58:53.891848328 +0000 UTC m=+754.191258187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls") pod "maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" (UID: "519f6640-0db9-43ac-a221-fd8c2def0a81") : secret "maas-api-e2e-derive-4705e99c-serving-cert" not found Oct 30 21:58:53.393751 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.392585 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519f6640-0db9-43ac-a221-fd8c2def0a81-openshift-service-ca\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.393839 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.391774 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519f6640-0db9-43ac-a221-fd8c2def0a81-openshift-service-ca\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.394237 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.394216 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-metrics-certs\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.399244 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.399222 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-metrics-certs\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.474405 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.471170 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tbk\" (UniqueName: \"kubernetes.io/projected/519f6640-0db9-43ac-a221-fd8c2def0a81-kube-api-access-r4tbk\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.576738 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.576696 4991 patch_prober.go:29] interesting pod/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:58:53.577021 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.576991 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:58:53.902193 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.902111 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:53.904827 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:53.904799 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls\") pod \"maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:54.088896 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:54.088860 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:54.514057 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:54.514027 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx"] Oct 30 21:58:54.527347 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:58:54.519692 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519f6640_0db9_43ac_a221_fd8c2def0a81.slice/crio-6d3cdc2ed826a676f5878e425e80bf2133b71ed7ed40f6eb752c2630a469511c WatchSource:0}: Error finding container 6d3cdc2ed826a676f5878e425e80bf2133b71ed7ed40f6eb752c2630a469511c: Status 404 returned error can't find the container with id 6d3cdc2ed826a676f5878e425e80bf2133b71ed7ed40f6eb752c2630a469511c Oct 30 21:58:54.575346 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:54.574640 4991 patch_prober.go:29] interesting pod/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:58:54.575346 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:54.574697 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:58:55.009645 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:55.009611 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" event={"ID":"519f6640-0db9-43ac-a221-fd8c2def0a81","Type":"ContainerStarted","Data":"6d3cdc2ed826a676f5878e425e80bf2133b71ed7ed40f6eb752c2630a469511c"} Oct 30 21:58:55.574000 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:55.573969 4991 patch_prober.go:29] interesting pod/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:58:55.574582 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:55.574550 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:58:56.017006 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:56.016973 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" event={"ID":"519f6640-0db9-43ac-a221-fd8c2def0a81","Type":"ContainerStarted","Data":"5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce"} Oct 30 21:58:56.017193 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:56.017036 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:58:56.172819 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:56.172772 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" podStartSLOduration=2.20790669 podStartE2EDuration="3.17275812s" totalImagesPullingTime="964.85143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:56.135772954 +0000 UTC m=+756.435182837" watchObservedRunningTime="2026-10-30 21:58:56.17275812 +0000 UTC m=+756.472167999" Oct 30 21:58:56.570184 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:56.570149 4991 patch_prober.go:29] interesting pod/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:58:56.570362 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:56.570205 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:58:57.573245 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:57.573209 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:57.573770 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:57.573654 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:58:57.574236 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:58:57.574214 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:59:00.046372 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:00.046340 4991 generic.go:356] "Generic (PLEG): container finished" podID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerID="2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a" exitCode=0 Oct 30 21:59:00.046795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:00.046401 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" event={"ID":"4fbd8d30-28e1-44d4-880c-25bcd3e67696","Type":"ContainerDied","Data":"2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a"} Oct 30 21:59:00.046795 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:00.046739 4991 scope.go:122] "RemoveContainer" containerID="2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a" Oct 30 21:59:00.048203 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:00.048165 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:00.048299 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:00.048257 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:00.048367 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:00.048305 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:01.054405 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:01.054370 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" event={"ID":"4fbd8d30-28e1-44d4-880c-25bcd3e67696","Type":"ContainerStarted","Data":"60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c"} Oct 30 21:59:01.899176 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:01.899140 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:59:01.900519 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:01.900495 4991 patch_prober.go:29] interesting pod/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:59:01.900635 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:01.900550 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:59:02.046119 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:02.046082 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 21:59:02.898692 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:02.898654 4991 patch_prober.go:29] interesting pod/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:59:02.899097 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:02.898713 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:59:03.694380 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:03.694344 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:59:03.898846 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:03.898811 4991 patch_prober.go:29] interesting pod/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:59:03.899252 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:03.898878 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:59:04.898494 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:04.898458 4991 patch_prober.go:29] interesting pod/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:59:04.898680 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:04.898508 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:59:05.898736 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:05.898701 4991 patch_prober.go:29] interesting pod/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:59:05.899209 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:05.898771 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:59:06.902344 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:06.902283 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:59:06.903442 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:06.903421 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:59:17.173067 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.173027 4991 generic.go:356] "Generic (PLEG): container finished" podID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerID="4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c" exitCode=0 Oct 30 21:59:17.173569 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.173098 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" event={"ID":"7bf81202-ac7f-478e-9819-2e3030e9bd78","Type":"ContainerDied","Data":"4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c"} Oct 30 21:59:17.173569 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.173504 4991 scope.go:122] "RemoveContainer" containerID="4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c" Oct 30 21:59:17.174705 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.174678 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:17.174797 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.174745 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:17.174797 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.174776 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:17.911735 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:17.911701 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:59:17.946166 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.946129 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx"] Oct 30 21:59:17.946430 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:17.946403 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" podUID="f7b606d7-cfb7-4f93-80ed-63c5f77802d5" containerName="maas-api" containerID="cri-o://b218afe023b7dd672e92285fc6b30ae3a5e366eb6381c8077c098dc8b9158f3d" gracePeriod=30 Oct 30 21:59:18.183889 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.183097 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" event={"ID":"7bf81202-ac7f-478e-9819-2e3030e9bd78","Type":"ContainerStarted","Data":"c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517"} Oct 30 21:59:18.185862 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.185837 4991 generic.go:356] "Generic (PLEG): container finished" podID="f7b606d7-cfb7-4f93-80ed-63c5f77802d5" containerID="b218afe023b7dd672e92285fc6b30ae3a5e366eb6381c8077c098dc8b9158f3d" exitCode=0 Oct 30 21:59:18.186022 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.185961 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" event={"ID":"f7b606d7-cfb7-4f93-80ed-63c5f77802d5","Type":"ContainerDied","Data":"b218afe023b7dd672e92285fc6b30ae3a5e366eb6381c8077c098dc8b9158f3d"} Oct 30 21:59:18.258236 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.258208 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:59:18.264818 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.264792 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-kube-api-access-8b987\" (UniqueName: \"kubernetes.io/projected/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-kube-api-access-8b987\") pod \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " Oct 30 21:59:18.264948 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.264849 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-openshift-service-ca\") pod \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " Oct 30 21:59:18.264948 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.264876 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls\") pod \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " Oct 30 21:59:18.265498 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.265203 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-openshift-service-ca" pod "f7b606d7-cfb7-4f93-80ed-63c5f77802d5" (UID: "f7b606d7-cfb7-4f93-80ed-63c5f77802d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:18.267153 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.267128 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-kube-api-access-8b987" pod "f7b606d7-cfb7-4f93-80ed-63c5f77802d5" (UID: "f7b606d7-cfb7-4f93-80ed-63c5f77802d5"). InnerVolumeSpecName "kube-api-access-8b987". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:18.267245 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.267192 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls" pod "f7b606d7-cfb7-4f93-80ed-63c5f77802d5" (UID: "f7b606d7-cfb7-4f93-80ed-63c5f77802d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:18.365693 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.365655 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-metrics-certs\") pod \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\" (UID: \"f7b606d7-cfb7-4f93-80ed-63c5f77802d5\") " Oct 30 21:59:18.365963 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.365945 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.366044 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.365968 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.366044 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.365983 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b987\" (UniqueName: \"kubernetes.io/projected/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-kube-api-access-8b987\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.367789 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.367761 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-metrics-certs" pod "f7b606d7-cfb7-4f93-80ed-63c5f77802d5" (UID: "f7b606d7-cfb7-4f93-80ed-63c5f77802d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:18.462702 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.462622 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:59:18.462961 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.462931 4991 patch_prober.go:29] interesting pod/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:59:18.463028 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.463005 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:59:18.466558 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:18.466539 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b606d7-cfb7-4f93-80ed-63c5f77802d5-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:19.192537 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.192507 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" Oct 30 21:59:19.192945 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.192540 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx" event={"ID":"f7b606d7-cfb7-4f93-80ed-63c5f77802d5","Type":"ContainerDied","Data":"b05c1704654f310fe03de8553e7d4dcec00da8d59c21a4257809ffc529fd637d"} Oct 30 21:59:19.192945 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.192600 4991 scope.go:122] "RemoveContainer" containerID="b218afe023b7dd672e92285fc6b30ae3a5e366eb6381c8077c098dc8b9158f3d" Oct 30 21:59:19.362541 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.362509 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx"] Oct 30 21:59:19.397054 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.397018 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63d97fec-6668946754-bggcx"] Oct 30 21:59:19.462877 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.462841 4991 patch_prober.go:29] interesting pod/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:59:19.463048 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.462896 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:59:19.876353 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:19.876297 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b606d7-cfb7-4f93-80ed-63c5f77802d5" path="/var/lib/kubelet/pods/f7b606d7-cfb7-4f93-80ed-63c5f77802d5/volumes" Oct 30 21:59:20.462942 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:20.462898 4991 patch_prober.go:29] interesting pod/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:59:20.463384 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:20.462980 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:59:21.462793 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:21.462760 4991 patch_prober.go:29] interesting pod/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:59:21.462980 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:21.462813 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:59:22.462822 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:22.462784 4991 patch_prober.go:29] interesting pod/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:59:22.462996 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:22.462839 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:59:23.467859 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:23.467825 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:59:23.468270 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:23.468142 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:59:23.468737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:23.468722 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 21:59:24.142341 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.139182 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh"] Oct 30 21:59:24.142341 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.140127 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" podUID="3a40511a-c02b-4b40-8c9b-f4a2af3a784c" containerName="maas-api" containerID="cri-o://def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2" gracePeriod=30 Oct 30 21:59:24.450671 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.450640 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:59:24.557013 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.556980 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls\") pod \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " Oct 30 21:59:24.557013 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.557021 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-metrics-certs\") pod \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " Oct 30 21:59:24.557588 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.557086 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-openshift-service-ca\") pod \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " Oct 30 21:59:24.557588 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.557305 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-kube-api-access-9n8t8\" (UniqueName: \"kubernetes.io/projected/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-kube-api-access-9n8t8\") pod \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\" (UID: \"3a40511a-c02b-4b40-8c9b-f4a2af3a784c\") " Oct 30 21:59:24.557683 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.557578 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-openshift-service-ca" pod "3a40511a-c02b-4b40-8c9b-f4a2af3a784c" (UID: "3a40511a-c02b-4b40-8c9b-f4a2af3a784c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:24.557771 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.557740 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:24.559559 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.559524 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-kube-api-access-9n8t8" pod "3a40511a-c02b-4b40-8c9b-f4a2af3a784c" (UID: "3a40511a-c02b-4b40-8c9b-f4a2af3a784c"). InnerVolumeSpecName "kube-api-access-9n8t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:24.559801 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.559783 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls" pod "3a40511a-c02b-4b40-8c9b-f4a2af3a784c" (UID: "3a40511a-c02b-4b40-8c9b-f4a2af3a784c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:24.559885 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.559826 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-metrics-certs" pod "3a40511a-c02b-4b40-8c9b-f4a2af3a784c" (UID: "3a40511a-c02b-4b40-8c9b-f4a2af3a784c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:24.659044 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.659002 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n8t8\" (UniqueName: \"kubernetes.io/projected/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-kube-api-access-9n8t8\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:24.659044 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.659044 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:24.659251 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:24.659060 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a40511a-c02b-4b40-8c9b-f4a2af3a784c-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.235309 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.235276 4991 generic.go:356] "Generic (PLEG): container finished" podID="3a40511a-c02b-4b40-8c9b-f4a2af3a784c" containerID="def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2" exitCode=0 Oct 30 21:59:25.235466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.235331 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" Oct 30 21:59:25.235466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.235362 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" event={"ID":"3a40511a-c02b-4b40-8c9b-f4a2af3a784c","Type":"ContainerDied","Data":"def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2"} Oct 30 21:59:25.235466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.235422 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh" event={"ID":"3a40511a-c02b-4b40-8c9b-f4a2af3a784c","Type":"ContainerDied","Data":"464c71de33b5c93ee14844ab29403af66d0df694d2bfcdca0b45f7562655e0b4"} Oct 30 21:59:25.235466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.235454 4991 scope.go:122] "RemoveContainer" containerID="def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2" Oct 30 21:59:25.245489 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.245473 4991 scope.go:122] "RemoveContainer" containerID="def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2" Oct 30 21:59:25.245717 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:25.245700 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2\": container with ID starting with def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2 not found: ID does not exist" containerID="def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2" Oct 30 21:59:25.245769 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.245724 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2"} err="failed to get container status \"def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2\": rpc error: code = NotFound desc = could not find container \"def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2\": container with ID starting with def9c0141188ad7a367b2c23a3ff17c24d34acd10cee49c74dcadea7021150e2 not found: ID does not exist" Oct 30 21:59:25.280757 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.280734 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54"] Oct 30 21:59:25.280947 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.280927 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" podUID="83e6a169-ba2b-4a5b-b904-21b34d262b01" containerName="maas-api" containerID="cri-o://1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba" gracePeriod=30 Oct 30 21:59:25.432963 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.432935 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh"] Oct 30 21:59:25.439528 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.439499 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b0026e9-656b6f9fbc-4k8rh"] Oct 30 21:59:25.548579 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.548554 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:59:25.567968 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.567942 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e6a169-ba2b-4a5b-b904-21b34d262b01-kube-api-access-hm2vw\" (UniqueName: \"kubernetes.io/projected/83e6a169-ba2b-4a5b-b904-21b34d262b01-kube-api-access-hm2vw\") pod \"83e6a169-ba2b-4a5b-b904-21b34d262b01\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " Oct 30 21:59:25.568400 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.568000 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls\") pod \"83e6a169-ba2b-4a5b-b904-21b34d262b01\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " Oct 30 21:59:25.568400 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.568159 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e6a169-ba2b-4a5b-b904-21b34d262b01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6a169-ba2b-4a5b-b904-21b34d262b01-openshift-service-ca\") pod \"83e6a169-ba2b-4a5b-b904-21b34d262b01\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " Oct 30 21:59:25.568400 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.568213 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-metrics-certs\") pod \"83e6a169-ba2b-4a5b-b904-21b34d262b01\" (UID: \"83e6a169-ba2b-4a5b-b904-21b34d262b01\") " Oct 30 21:59:25.568637 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.568548 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e6a169-ba2b-4a5b-b904-21b34d262b01-openshift-service-ca" pod "83e6a169-ba2b-4a5b-b904-21b34d262b01" (UID: "83e6a169-ba2b-4a5b-b904-21b34d262b01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:25.569021 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.569000 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6a169-ba2b-4a5b-b904-21b34d262b01-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.570415 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.570389 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e6a169-ba2b-4a5b-b904-21b34d262b01-kube-api-access-hm2vw" pod "83e6a169-ba2b-4a5b-b904-21b34d262b01" (UID: "83e6a169-ba2b-4a5b-b904-21b34d262b01"). InnerVolumeSpecName "kube-api-access-hm2vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:25.570728 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.570700 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-metrics-certs" pod "83e6a169-ba2b-4a5b-b904-21b34d262b01" (UID: "83e6a169-ba2b-4a5b-b904-21b34d262b01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:25.570822 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.570792 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls" pod "83e6a169-ba2b-4a5b-b904-21b34d262b01" (UID: "83e6a169-ba2b-4a5b-b904-21b34d262b01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:25.669936 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.669908 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm2vw\" (UniqueName: \"kubernetes.io/projected/83e6a169-ba2b-4a5b-b904-21b34d262b01-kube-api-access-hm2vw\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.669936 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.669936 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.670099 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.669947 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e6a169-ba2b-4a5b-b904-21b34d262b01-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.875198 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:25.875117 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a40511a-c02b-4b40-8c9b-f4a2af3a784c" path="/var/lib/kubelet/pods/3a40511a-c02b-4b40-8c9b-f4a2af3a784c/volumes" Oct 30 21:59:26.243233 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.243201 4991 generic.go:356] "Generic (PLEG): container finished" podID="83e6a169-ba2b-4a5b-b904-21b34d262b01" containerID="1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba" exitCode=0 Oct 30 21:59:26.243420 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.243254 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" Oct 30 21:59:26.243420 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.243284 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" event={"ID":"83e6a169-ba2b-4a5b-b904-21b34d262b01","Type":"ContainerDied","Data":"1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba"} Oct 30 21:59:26.243420 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.243350 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54" event={"ID":"83e6a169-ba2b-4a5b-b904-21b34d262b01","Type":"ContainerDied","Data":"07d051ebb135ac26c50586a603696550a95bb55f43a1eb0135faf911175d040f"} Oct 30 21:59:26.243420 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.243380 4991 scope.go:122] "RemoveContainer" containerID="1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba" Oct 30 21:59:26.252682 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.252654 4991 scope.go:122] "RemoveContainer" containerID="1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba" Oct 30 21:59:26.252974 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:26.252956 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba\": container with ID starting with 1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba not found: ID does not exist" containerID="1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba" Oct 30 21:59:26.253032 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.252983 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba"} err="failed to get container status \"1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba\": rpc error: code = NotFound desc = could not find container \"1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba\": container with ID starting with 1f7e509d115ea4e4d61534689a196db1db86b0cb8c3f4d1c222a90f1412b12ba not found: ID does not exist" Oct 30 21:59:26.431708 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.431675 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54"] Oct 30 21:59:26.478389 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:26.478356 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b2e37ae-866589d9dc-r9t54"] Oct 30 21:59:27.878061 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:27.878023 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e6a169-ba2b-4a5b-b904-21b34d262b01" path="/var/lib/kubelet/pods/83e6a169-ba2b-4a5b-b904-21b34d262b01/volumes" Oct 30 21:59:27.905660 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:27.905631 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9"] Oct 30 21:59:27.905869 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:27.905847 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" containerID="cri-o://366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a" gracePeriod=30 Oct 30 21:59:33.254805 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.254780 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:59:33.289743 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.289717 4991 generic.go:356] "Generic (PLEG): container finished" podID="849162f4-291a-445d-965d-63579ecc637b" containerID="366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a" exitCode=0 Oct 30 21:59:33.289873 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.289771 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" Oct 30 21:59:33.289873 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.289797 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" event={"ID":"849162f4-291a-445d-965d-63579ecc637b","Type":"ContainerDied","Data":"366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a"} Oct 30 21:59:33.289873 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.289850 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9" event={"ID":"849162f4-291a-445d-965d-63579ecc637b","Type":"ContainerDied","Data":"748307d1f9598a15da94dcd46c715c69e45b7aaca82ff03137c298856e0e2feb"} Oct 30 21:59:33.289873 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.289873 4991 scope.go:122] "RemoveContainer" containerID="366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a" Oct 30 21:59:33.299005 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.298987 4991 scope.go:122] "RemoveContainer" containerID="c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4" Oct 30 21:59:33.307219 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.307204 4991 scope.go:122] "RemoveContainer" containerID="366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a" Oct 30 21:59:33.307489 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:33.307473 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a\": container with ID starting with 366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a not found: ID does not exist" containerID="366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a" Oct 30 21:59:33.307532 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.307500 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a"} err="failed to get container status \"366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a\": rpc error: code = NotFound desc = could not find container \"366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a\": container with ID starting with 366cd2d061d51a89304fb093dd2f8ebe2e3e302965c56a30138fe956a05b4d5a not found: ID does not exist" Oct 30 21:59:33.307532 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.307518 4991 scope.go:122] "RemoveContainer" containerID="c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4" Oct 30 21:59:33.307730 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:33.307711 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4\": container with ID starting with c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4 not found: ID does not exist" containerID="c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4" Oct 30 21:59:33.307793 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.307740 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4"} err="failed to get container status \"c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4\": rpc error: code = NotFound desc = could not find container \"c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4\": container with ID starting with c85f5465595a66091e594faa01dfa423a1464590c92460cde4e491a0df9043c4 not found: ID does not exist" Oct 30 21:59:33.378844 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.378818 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-socket\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.378972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.378878 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849162f4-291a-445d-965d-63579ecc637b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849162f4-291a-445d-965d-63579ecc637b-istiod-ca-cert\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.378972 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.378950 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-data\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.379104 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379046 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-envoy\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.379166 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379102 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-credential-socket\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.379166 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379137 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/849162f4-291a-445d-965d-63579ecc637b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849162f4-291a-445d-965d-63579ecc637b-istio-podinfo\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.379266 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379155 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-socket" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:33.379266 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379174 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-kube-api-access-txb6k\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-kube-api-access-txb6k\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.379266 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379213 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-certs\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.379266 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379245 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-istio-token\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-istio-token\") pod \"849162f4-291a-445d-965d-63579ecc637b\" (UID: \"849162f4-291a-445d-965d-63579ecc637b\") " Oct 30 21:59:33.379533 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379452 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849162f4-291a-445d-965d-63579ecc637b-istiod-ca-cert" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:33.379686 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379664 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.379774 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379691 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849162f4-291a-445d-965d-63579ecc637b-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.379774 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379722 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-data" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:33.379961 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.379924 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-certs" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:33.380129 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.380108 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-credential-socket" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:33.382003 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.381982 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-envoy" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:33.382729 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.382627 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/849162f4-291a-445d-965d-63579ecc637b-istio-podinfo" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:33.382729 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.382694 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-istio-token" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:33.382916 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.382787 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-kube-api-access-txb6k" pod "849162f4-291a-445d-965d-63579ecc637b" (UID: "849162f4-291a-445d-965d-63579ecc637b"). InnerVolumeSpecName "kube-api-access-txb6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:33.480733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.480658 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.480733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.480684 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.480733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.480694 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.480733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.480703 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849162f4-291a-445d-965d-63579ecc637b-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.480733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.480711 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txb6k\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-kube-api-access-txb6k\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.480733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.480721 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849162f4-291a-445d-965d-63579ecc637b-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.480733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.480730 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849162f4-291a-445d-965d-63579ecc637b-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.705565 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.705532 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9"] Oct 30 21:59:33.710038 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.710012 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4b2e37ae-gw-openshift-default-b4b4489bf-vmtx9"] Oct 30 21:59:33.874190 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:33.874105 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849162f4-291a-445d-965d-63579ecc637b" path="/var/lib/kubelet/pods/849162f4-291a-445d-965d-63579ecc637b/volumes" Oct 30 21:59:41.684244 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684208 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t"] Oct 30 21:59:41.684691 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684675 4991 cpu_manager.go:414] "containerMap: removing container" podUID="83e6a169-ba2b-4a5b-b904-21b34d262b01" containerName="maas-api" Oct 30 21:59:41.684737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684692 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e6a169-ba2b-4a5b-b904-21b34d262b01" containerName="maas-api" Oct 30 21:59:41.684737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684708 4991 cpu_manager.go:414] "containerMap: removing container" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" Oct 30 21:59:41.684737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684713 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" Oct 30 21:59:41.684737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684720 4991 cpu_manager.go:414] "containerMap: removing container" podUID="3a40511a-c02b-4b40-8c9b-f4a2af3a784c" containerName="maas-api" Oct 30 21:59:41.684737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684726 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a40511a-c02b-4b40-8c9b-f4a2af3a784c" containerName="maas-api" Oct 30 21:59:41.684737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684735 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f7b606d7-cfb7-4f93-80ed-63c5f77802d5" containerName="maas-api" Oct 30 21:59:41.684737 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684739 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b606d7-cfb7-4f93-80ed-63c5f77802d5" containerName="maas-api" Oct 30 21:59:41.684934 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684759 4991 cpu_manager.go:414] "containerMap: removing container" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" Oct 30 21:59:41.684934 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684764 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" Oct 30 21:59:41.684934 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684852 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b606d7-cfb7-4f93-80ed-63c5f77802d5" containerName="maas-api" Oct 30 21:59:41.684934 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684864 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" Oct 30 21:59:41.684934 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684873 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e6a169-ba2b-4a5b-b904-21b34d262b01" containerName="maas-api" Oct 30 21:59:41.684934 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.684884 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a40511a-c02b-4b40-8c9b-f4a2af3a784c" containerName="maas-api" Oct 30 21:59:41.689175 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.689157 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.744673 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.744640 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t"] Oct 30 21:59:41.762832 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.762799 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.762979 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.762861 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5ff2f-96f2-4477-b319-219840d3db21-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.762979 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.762925 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xm4\" (UniqueName: \"kubernetes.io/projected/a5d5ff2f-96f2-4477-b319-219840d3db21-kube-api-access-v2xm4\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.763097 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.763072 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-metrics-certs\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.787553 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.787526 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-7f66ab-serving-cert\"" Oct 30 21:59:41.864405 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.864305 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5ff2f-96f2-4477-b319-219840d3db21-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.864548 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.864423 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xm4\" (UniqueName: \"kubernetes.io/projected/a5d5ff2f-96f2-4477-b319-219840d3db21-kube-api-access-v2xm4\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.864548 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.864503 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-metrics-certs\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.864622 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.864583 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.864702 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:41.864687 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-serving-cert: secret "maas-api-e2e-worker-w2-7f66ab-serving-cert" not found Oct 30 21:59:41.864769 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:41.864754 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls podName:a5d5ff2f-96f2-4477-b319-219840d3db21 nodeName:}" failed. No retries permitted until 2026-10-30 21:59:42.364735231 +0000 UTC m=+802.664145089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls") pod "maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" (UID: "a5d5ff2f-96f2-4477-b319-219840d3db21") : secret "maas-api-e2e-worker-w2-7f66ab-serving-cert" not found Oct 30 21:59:41.865029 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.865009 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5ff2f-96f2-4477-b319-219840d3db21-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.866916 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.866899 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-metrics-certs\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:41.924494 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:41.924468 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xm4\" (UniqueName: \"kubernetes.io/projected/a5d5ff2f-96f2-4477-b319-219840d3db21-kube-api-access-v2xm4\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:42.370573 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:42.370540 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:42.372984 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:42.372953 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:42.605560 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:42.605535 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:42.997472 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:42.997443 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t"] Oct 30 21:59:43.003075 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:59:43.003046 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d5ff2f_96f2_4477_b319_219840d3db21.slice/crio-c4e3f2da5994537d6dc967abcdc7d1e4b6fcbb598689b0c36d20ddf02a83ab4c WatchSource:0}: Error finding container c4e3f2da5994537d6dc967abcdc7d1e4b6fcbb598689b0c36d20ddf02a83ab4c: Status 404 returned error can't find the container with id c4e3f2da5994537d6dc967abcdc7d1e4b6fcbb598689b0c36d20ddf02a83ab4c Oct 30 21:59:43.353202 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:43.353120 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" event={"ID":"a5d5ff2f-96f2-4477-b319-219840d3db21","Type":"ContainerStarted","Data":"c4e3f2da5994537d6dc967abcdc7d1e4b6fcbb598689b0c36d20ddf02a83ab4c"} Oct 30 21:59:44.367796 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:44.367740 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" event={"ID":"a5d5ff2f-96f2-4477-b319-219840d3db21","Type":"ContainerStarted","Data":"57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24"} Oct 30 21:59:44.368230 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:44.367865 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:44.479428 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:44.479371 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" podStartSLOduration=2.545138018 podStartE2EDuration="3.479354306s" totalImagesPullingTime="934.216288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:44.479069615 +0000 UTC m=+804.778479492" watchObservedRunningTime="2026-10-30 21:59:44.479354306 +0000 UTC m=+804.778764187" Oct 30 21:59:45.930896 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:45.930866 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j"] Oct 30 21:59:45.931346 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:45.931033 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" podUID="4ff15627-966f-4f5b-85f3-272af51a24da" containerName="maas-api" containerID="cri-o://8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2" gracePeriod=30 Oct 30 21:59:46.199335 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.199290 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:59:46.324989 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.324959 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff15627-966f-4f5b-85f3-272af51a24da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff15627-966f-4f5b-85f3-272af51a24da-openshift-service-ca\") pod \"4ff15627-966f-4f5b-85f3-272af51a24da\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " Oct 30 21:59:46.325177 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.325074 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-metrics-certs\") pod \"4ff15627-966f-4f5b-85f3-272af51a24da\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " Oct 30 21:59:46.325177 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.325153 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff15627-966f-4f5b-85f3-272af51a24da-kube-api-access-n2wvz\" (UniqueName: \"kubernetes.io/projected/4ff15627-966f-4f5b-85f3-272af51a24da-kube-api-access-n2wvz\") pod \"4ff15627-966f-4f5b-85f3-272af51a24da\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " Oct 30 21:59:46.325293 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.325238 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls\") pod \"4ff15627-966f-4f5b-85f3-272af51a24da\" (UID: \"4ff15627-966f-4f5b-85f3-272af51a24da\") " Oct 30 21:59:46.325640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.325389 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff15627-966f-4f5b-85f3-272af51a24da-openshift-service-ca" pod "4ff15627-966f-4f5b-85f3-272af51a24da" (UID: "4ff15627-966f-4f5b-85f3-272af51a24da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:46.325640 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.325608 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff15627-966f-4f5b-85f3-272af51a24da-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:46.327445 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.327373 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-metrics-certs" pod "4ff15627-966f-4f5b-85f3-272af51a24da" (UID: "4ff15627-966f-4f5b-85f3-272af51a24da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:46.327989 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.327962 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff15627-966f-4f5b-85f3-272af51a24da-kube-api-access-n2wvz" pod "4ff15627-966f-4f5b-85f3-272af51a24da" (UID: "4ff15627-966f-4f5b-85f3-272af51a24da"). InnerVolumeSpecName "kube-api-access-n2wvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:46.328439 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.328413 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls" pod "4ff15627-966f-4f5b-85f3-272af51a24da" (UID: "4ff15627-966f-4f5b-85f3-272af51a24da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:46.380290 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.380260 4991 generic.go:356] "Generic (PLEG): container finished" podID="4ff15627-966f-4f5b-85f3-272af51a24da" containerID="8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2" exitCode=0 Oct 30 21:59:46.380441 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.380300 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" Oct 30 21:59:46.380441 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.380345 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" event={"ID":"4ff15627-966f-4f5b-85f3-272af51a24da","Type":"ContainerDied","Data":"8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2"} Oct 30 21:59:46.380441 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.380395 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j" event={"ID":"4ff15627-966f-4f5b-85f3-272af51a24da","Type":"ContainerDied","Data":"608dae5826244dad5ce3b4f83a1754a384aa9745a83d44971f17160d672e86b8"} Oct 30 21:59:46.380441 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.380419 4991 scope.go:122] "RemoveContainer" containerID="8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2" Oct 30 21:59:46.390220 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.390203 4991 scope.go:122] "RemoveContainer" containerID="8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2" Oct 30 21:59:46.390523 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:46.390500 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2\": container with ID starting with 8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2 not found: ID does not exist" containerID="8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2" Oct 30 21:59:46.390576 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.390531 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2"} err="failed to get container status \"8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2\": rpc error: code = NotFound desc = could not find container \"8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2\": container with ID starting with 8ce4c3a136328b4628fd90943ffa552bc37352076bc26e3f001b6c51ff7b02d2 not found: ID does not exist" Oct 30 21:59:46.432419 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.432348 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:46.432419 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.432373 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff15627-966f-4f5b-85f3-272af51a24da-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:46.432419 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.432384 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2wvz\" (UniqueName: \"kubernetes.io/projected/4ff15627-966f-4f5b-85f3-272af51a24da-kube-api-access-n2wvz\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:46.537376 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.537349 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j"] Oct 30 21:59:46.539863 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:46.539840 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a41d4db-7bff979b87-ccr5j"] Oct 30 21:59:47.873643 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:47.873614 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff15627-966f-4f5b-85f3-272af51a24da" path="/var/lib/kubelet/pods/4ff15627-966f-4f5b-85f3-272af51a24da/volumes" Oct 30 21:59:47.910801 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:47.910773 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 21:59:49.532457 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:49.532420 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z"] Oct 30 21:59:49.532834 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:49.532701 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" containerID="cri-o://60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c" gracePeriod=30 Oct 30 21:59:50.378068 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:50.378040 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 21:59:50.976025 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:50.975995 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp"] Oct 30 21:59:50.976446 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:50.976438 4991 cpu_manager.go:414] "containerMap: removing container" podUID="4ff15627-966f-4f5b-85f3-272af51a24da" containerName="maas-api" Oct 30 21:59:50.976489 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:50.976451 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff15627-966f-4f5b-85f3-272af51a24da" containerName="maas-api" Oct 30 21:59:50.976553 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:50.976543 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="849162f4-291a-445d-965d-63579ecc637b" containerName="istio-proxy" Oct 30 21:59:50.976588 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:50.976555 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff15627-966f-4f5b-85f3-272af51a24da" containerName="maas-api" Oct 30 21:59:50.981253 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:50.981230 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.044346 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.032722 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e524ce-openshift-default-dockercfg-zs996\"" Oct 30 21:59:51.082685 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.082645 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp"] Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170176 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170258 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170367 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13dca42e-3bfb-4105-a04b-f5c813540911-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170419 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170457 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170490 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfzv\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-kube-api-access-zmfzv\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170525 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170560 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.172373 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.170689 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13dca42e-3bfb-4105-a04b-f5c813540911-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271562 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271467 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13dca42e-3bfb-4105-a04b-f5c813540911-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271562 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271532 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271580 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271609 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13dca42e-3bfb-4105-a04b-f5c813540911-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271640 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271668 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271701 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfzv\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-kube-api-access-zmfzv\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271734 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.271800 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.271765 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.272162 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.272018 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.272162 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.272100 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.272281 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.272262 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.272692 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.272670 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13dca42e-3bfb-4105-a04b-f5c813540911-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.272863 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.272846 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.274097 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.274072 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13dca42e-3bfb-4105-a04b-f5c813540911-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.274521 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.274504 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.335118 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.335083 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfzv\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-kube-api-access-zmfzv\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.335781 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.335761 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.594877 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.594796 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:51.904688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.904602 4991 patch_prober.go:29] interesting pod/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:59:51.904688 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:51.904675 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:59:52.084802 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.084770 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp"] Oct 30 21:59:52.090438 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:59:52.090410 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dca42e_3bfb_4105_a04b_f5c813540911.slice/crio-99e6abada3c96635e31a35a1441fcd2e95fe40e427547b2885e62cca106e703f WatchSource:0}: Error finding container 99e6abada3c96635e31a35a1441fcd2e95fe40e427547b2885e62cca106e703f: Status 404 returned error can't find the container with id 99e6abada3c96635e31a35a1441fcd2e95fe40e427547b2885e62cca106e703f Oct 30 21:59:52.092546 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.092517 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:52.092617 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.092577 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:52.092617 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.092609 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:59:52.392098 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.392053 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw"] Oct 30 21:59:52.396598 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.396575 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.418879 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.418810 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" event={"ID":"13dca42e-3bfb-4105-a04b-f5c813540911","Type":"ContainerStarted","Data":"fd2fdf38e1a0d9e52f76e677a6890c578303f5f25385c4b4f96dc9122ac805bc"} Oct 30 21:59:52.418879 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.418865 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" event={"ID":"13dca42e-3bfb-4105-a04b-f5c813540911","Type":"ContainerStarted","Data":"99e6abada3c96635e31a35a1441fcd2e95fe40e427547b2885e62cca106e703f"} Oct 30 21:59:52.443217 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.443191 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw"] Oct 30 21:59:52.445913 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.445887 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b47e8d-serving-cert\"" Oct 30 21:59:52.482563 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.482534 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.482656 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.482577 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c7nl\" (UniqueName: \"kubernetes.io/projected/9194f4ba-5aea-450b-8ac3-efefa4b1e743-kube-api-access-6c7nl\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.482782 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.482761 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9194f4ba-5aea-450b-8ac3-efefa4b1e743-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.482849 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.482833 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-metrics-certs\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.584299 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.584264 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c7nl\" (UniqueName: \"kubernetes.io/projected/9194f4ba-5aea-450b-8ac3-efefa4b1e743-kube-api-access-6c7nl\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.584504 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.584358 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9194f4ba-5aea-450b-8ac3-efefa4b1e743-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.584504 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.584389 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-metrics-certs\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.584504 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.584454 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.584680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:52.584559 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-serving-cert: secret "maas-api-e2e-worker-w3-b47e8d-serving-cert" not found Oct 30 21:59:52.584680 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:52.584619 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls podName:9194f4ba-5aea-450b-8ac3-efefa4b1e743 nodeName:}" failed. No retries permitted until 2026-10-30 21:59:53.084599554 +0000 UTC m=+813.384009428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls") pod "maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" (UID: "9194f4ba-5aea-450b-8ac3-efefa4b1e743") : secret "maas-api-e2e-worker-w3-b47e8d-serving-cert" not found Oct 30 21:59:52.585136 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.585111 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9194f4ba-5aea-450b-8ac3-efefa4b1e743-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.587035 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.587008 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-metrics-certs\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:52.595733 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.595711 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:52.636935 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.636111 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podStartSLOduration=2.636092004 podStartE2EDuration="2.636092004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:52.632202393 +0000 UTC m=+812.931612296" watchObservedRunningTime="2026-10-30 21:59:52.636092004 +0000 UTC m=+812.935501885" Oct 30 21:59:52.655939 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:52.655912 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c7nl\" (UniqueName: \"kubernetes.io/projected/9194f4ba-5aea-450b-8ac3-efefa4b1e743-kube-api-access-6c7nl\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:53.088838 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:53.088807 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:53.091240 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:53.091213 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:53.311982 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:53.311945 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:53.600429 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:53.600398 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:53.697706 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:59:53.697676 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9194f4ba_5aea_450b_8ac3_efefa4b1e743.slice/crio-ad99e93399902542c599421ae658004b2d674155e19df96267341b5de5c4f421 WatchSource:0}: Error finding container ad99e93399902542c599421ae658004b2d674155e19df96267341b5de5c4f421: Status 404 returned error can't find the container with id ad99e93399902542c599421ae658004b2d674155e19df96267341b5de5c4f421 Oct 30 21:59:53.700285 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:53.700200 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw"] Oct 30 21:59:54.431515 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.431481 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" event={"ID":"9194f4ba-5aea-450b-8ac3-efefa4b1e743","Type":"ContainerStarted","Data":"ad99e93399902542c599421ae658004b2d674155e19df96267341b5de5c4f421"} Oct 30 21:59:54.431886 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.431660 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:54.432469 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.432451 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 21:59:54.791775 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.791750 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:59:54.808746 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.808715 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-podinfo\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.808827 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.808760 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-certs\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.808905 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.808884 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-data\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.808955 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.808932 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-credential-socket\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.809025 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809012 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-kube-api-access-7g7ms\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-kube-api-access-7g7ms\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.809084 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809053 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-envoy\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.809084 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809077 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-socket\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809007 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-certs" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809118 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istiod-ca-cert\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809168 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-token\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-token\") pod \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\" (UID: \"4fbd8d30-28e1-44d4-880c-25bcd3e67696\") " Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809174 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-data" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809228 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-credential-socket" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809368 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-socket" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809506 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istiod-ca-cert" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809520 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809536 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809552 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.809600 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.809570 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.811113 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.811090 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-envoy" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:54.811202 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.811155 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-token" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:54.811350 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.811314 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-podinfo" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:54.811457 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.811364 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-kube-api-access-7g7ms" pod "4fbd8d30-28e1-44d4-880c-25bcd3e67696" (UID: "4fbd8d30-28e1-44d4-880c-25bcd3e67696"). InnerVolumeSpecName "kube-api-access-7g7ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:54.910632 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.910600 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g7ms\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-kube-api-access-7g7ms\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.910784 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.910631 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.910784 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.910680 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.910784 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.910695 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.910784 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:54.910738 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbd8d30-28e1-44d4-880c-25bcd3e67696-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:55.439193 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.439159 4991 generic.go:356] "Generic (PLEG): container finished" podID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerID="60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c" exitCode=0 Oct 30 21:59:55.439687 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.439208 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" Oct 30 21:59:55.439687 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.439240 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" event={"ID":"4fbd8d30-28e1-44d4-880c-25bcd3e67696","Type":"ContainerDied","Data":"60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c"} Oct 30 21:59:55.439687 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.439301 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z" event={"ID":"4fbd8d30-28e1-44d4-880c-25bcd3e67696","Type":"ContainerDied","Data":"dbc1ce71586dc6813c9e49a794096d4da40d16fcd3a5d5ba19e865f1aaaee97d"} Oct 30 21:59:55.439687 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.439338 4991 scope.go:122] "RemoveContainer" containerID="60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c" Oct 30 21:59:55.441014 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.440983 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" event={"ID":"9194f4ba-5aea-450b-8ac3-efefa4b1e743","Type":"ContainerStarted","Data":"8584b01fa4de5921d4aa1505c8b0b3017a72ee11407b9f7896b9f65417ea689f"} Oct 30 21:59:55.441128 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.441079 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 21:59:55.449608 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.449594 4991 scope.go:122] "RemoveContainer" containerID="2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a" Oct 30 21:59:55.458173 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.458157 4991 scope.go:122] "RemoveContainer" containerID="60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c" Oct 30 21:59:55.458491 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:55.458469 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c\": container with ID starting with 60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c not found: ID does not exist" containerID="60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c" Oct 30 21:59:55.458563 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.458502 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c"} err="failed to get container status \"60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c\": rpc error: code = NotFound desc = could not find container \"60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c\": container with ID starting with 60ca95ddf7e360a9d5008881611ad5dd0f0c10a501045dd8a3c4570c9718141c not found: ID does not exist" Oct 30 21:59:55.458563 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.458521 4991 scope.go:122] "RemoveContainer" containerID="2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a" Oct 30 21:59:55.458785 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:55.458767 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a\": container with ID starting with 2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a not found: ID does not exist" containerID="2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a" Oct 30 21:59:55.458841 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.458791 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a"} err="failed to get container status \"2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a\": rpc error: code = NotFound desc = could not find container \"2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a\": container with ID starting with 2d56189fcfea7770310f166a04d9d215ede0284bd652ec5a09c4b7d76db06b8a not found: ID does not exist" Oct 30 21:59:55.603033 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.602982 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" podStartSLOduration=2.6925267440000002 podStartE2EDuration="3.60296738s" totalImagesPullingTime="910.440636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:55.600776139 +0000 UTC m=+815.900186019" watchObservedRunningTime="2026-10-30 21:59:55.60296738 +0000 UTC m=+815.902377260" Oct 30 21:59:55.710466 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.710380 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z"] Oct 30 21:59:55.716456 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.716430 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2a41d4db-openshift-default-5dd47c8b4f-bwh6z"] Oct 30 21:59:55.875410 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:55.875378 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" path="/var/lib/kubelet/pods/4fbd8d30-28e1-44d4-880c-25bcd3e67696/volumes" Oct 30 21:59:57.484182 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.484148 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985"] Oct 30 21:59:57.484636 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.484623 4991 cpu_manager.go:414] "containerMap: removing container" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" Oct 30 21:59:57.484679 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.484638 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" Oct 30 21:59:57.484738 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.484729 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" Oct 30 21:59:57.484772 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.484743 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" Oct 30 21:59:57.489551 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.489530 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.537593 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.537567 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm4j\" (UniqueName: \"kubernetes.io/projected/b639357e-4f63-439a-b1b6-2e32bfa296a2-kube-api-access-zjm4j\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.537697 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.537601 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-metrics-certs\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.537697 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.537641 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.537766 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.537748 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b639357e-4f63-439a-b1b6-2e32bfa296a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.550125 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.550100 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985"] Oct 30 21:59:57.593142 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.593120 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-d09fff-serving-cert\"" Oct 30 21:59:57.638683 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.638652 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm4j\" (UniqueName: \"kubernetes.io/projected/b639357e-4f63-439a-b1b6-2e32bfa296a2-kube-api-access-zjm4j\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.638835 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.638698 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-metrics-certs\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.638835 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.638760 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.638835 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.638815 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b639357e-4f63-439a-b1b6-2e32bfa296a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.639861 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:57.639611 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-serving-cert: secret "maas-api-e2e-worker-w0-d09fff-serving-cert" not found Oct 30 21:59:57.639861 ip-10-0-137-17 kubenswrapper[4991]: E1030 21:59:57.639692 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls podName:b639357e-4f63-439a-b1b6-2e32bfa296a2 nodeName:}" failed. No retries permitted until 2026-10-30 21:59:58.139671792 +0000 UTC m=+818.439081664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls") pod "maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" (UID: "b639357e-4f63-439a-b1b6-2e32bfa296a2") : secret "maas-api-e2e-worker-w0-d09fff-serving-cert" not found Oct 30 21:59:57.640918 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.640885 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b639357e-4f63-439a-b1b6-2e32bfa296a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.642762 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.642739 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-metrics-certs\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:57.697467 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:57.697443 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm4j\" (UniqueName: \"kubernetes.io/projected/b639357e-4f63-439a-b1b6-2e32bfa296a2-kube-api-access-zjm4j\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:58.143760 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:58.143730 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:58.145984 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:58.145956 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:58.402250 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:58.402168 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 21:59:58.782940 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:58.782912 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985"] Oct 30 21:59:58.788241 ip-10-0-137-17 kubenswrapper[4991]: W1030 21:59:58.788203 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb639357e_4f63_439a_b1b6_2e32bfa296a2.slice/crio-9301942c6eba1c91e73975b613995c6198ed39507c91736bdc3e2a9878e84767 WatchSource:0}: Error finding container 9301942c6eba1c91e73975b613995c6198ed39507c91736bdc3e2a9878e84767: Status 404 returned error can't find the container with id 9301942c6eba1c91e73975b613995c6198ed39507c91736bdc3e2a9878e84767 Oct 30 21:59:59.466709 ip-10-0-137-17 kubenswrapper[4991]: I1030 21:59:59.466670 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" event={"ID":"b639357e-4f63-439a-b1b6-2e32bfa296a2","Type":"ContainerStarted","Data":"9301942c6eba1c91e73975b613995c6198ed39507c91736bdc3e2a9878e84767"} Oct 30 22:00:00.473982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:00.473938 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" event={"ID":"b639357e-4f63-439a-b1b6-2e32bfa296a2","Type":"ContainerStarted","Data":"226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6"} Oct 30 22:00:00.474382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:00.474093 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 22:00:00.615643 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:00.615581 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" podStartSLOduration=2.706561237 podStartE2EDuration="3.615566774s" totalImagesPullingTime="909.005537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:00.614937446 +0000 UTC m=+820.914347327" watchObservedRunningTime="2026-10-30 22:00:00.615566774 +0000 UTC m=+820.914976676" Oct 30 22:00:01.453711 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:01.453681 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 22:00:02.009219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.009179 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx"] Oct 30 22:00:02.009706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.009462 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" podUID="519f6640-0db9-43ac-a221-fd8c2def0a81" containerName="maas-api" containerID="cri-o://5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce" gracePeriod=30 Oct 30 22:00:02.025444 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.025301 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" podUID="519f6640-0db9-43ac-a221-fd8c2def0a81" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.67:8443/health\": dial tcp 10.132.0.67:8443: connect: connection refused" Oct 30 22:00:02.276544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.276475 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 22:00:02.400803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.400772 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519f6640-0db9-43ac-a221-fd8c2def0a81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519f6640-0db9-43ac-a221-fd8c2def0a81-openshift-service-ca\") pod \"519f6640-0db9-43ac-a221-fd8c2def0a81\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " Oct 30 22:00:02.401017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.400910 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-metrics-certs\") pod \"519f6640-0db9-43ac-a221-fd8c2def0a81\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " Oct 30 22:00:02.401017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.400948 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519f6640-0db9-43ac-a221-fd8c2def0a81-kube-api-access-r4tbk\" (UniqueName: \"kubernetes.io/projected/519f6640-0db9-43ac-a221-fd8c2def0a81-kube-api-access-r4tbk\") pod \"519f6640-0db9-43ac-a221-fd8c2def0a81\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " Oct 30 22:00:02.401142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.401025 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls\") pod \"519f6640-0db9-43ac-a221-fd8c2def0a81\" (UID: \"519f6640-0db9-43ac-a221-fd8c2def0a81\") " Oct 30 22:00:02.401199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.401172 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519f6640-0db9-43ac-a221-fd8c2def0a81-openshift-service-ca" pod "519f6640-0db9-43ac-a221-fd8c2def0a81" (UID: "519f6640-0db9-43ac-a221-fd8c2def0a81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:02.401474 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.401446 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519f6640-0db9-43ac-a221-fd8c2def0a81-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.406355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.402996 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519f6640-0db9-43ac-a221-fd8c2def0a81-kube-api-access-r4tbk" pod "519f6640-0db9-43ac-a221-fd8c2def0a81" (UID: "519f6640-0db9-43ac-a221-fd8c2def0a81"). InnerVolumeSpecName "kube-api-access-r4tbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:02.406355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.403198 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-metrics-certs" pod "519f6640-0db9-43ac-a221-fd8c2def0a81" (UID: "519f6640-0db9-43ac-a221-fd8c2def0a81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:02.406355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.403494 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls" pod "519f6640-0db9-43ac-a221-fd8c2def0a81" (UID: "519f6640-0db9-43ac-a221-fd8c2def0a81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:02.493463 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.493426 4991 generic.go:356] "Generic (PLEG): container finished" podID="519f6640-0db9-43ac-a221-fd8c2def0a81" containerID="5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce" exitCode=0 Oct 30 22:00:02.493662 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.493469 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" Oct 30 22:00:02.493662 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.493514 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" event={"ID":"519f6640-0db9-43ac-a221-fd8c2def0a81","Type":"ContainerDied","Data":"5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce"} Oct 30 22:00:02.493662 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.493573 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx" event={"ID":"519f6640-0db9-43ac-a221-fd8c2def0a81","Type":"ContainerDied","Data":"6d3cdc2ed826a676f5878e425e80bf2133b71ed7ed40f6eb752c2630a469511c"} Oct 30 22:00:02.493662 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.493603 4991 scope.go:122] "RemoveContainer" containerID="5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce" Oct 30 22:00:02.502209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.502187 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.502303 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.502214 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4tbk\" (UniqueName: \"kubernetes.io/projected/519f6640-0db9-43ac-a221-fd8c2def0a81-kube-api-access-r4tbk\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.502303 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.502230 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519f6640-0db9-43ac-a221-fd8c2def0a81-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.503846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.503831 4991 scope.go:122] "RemoveContainer" containerID="5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce" Oct 30 22:00:02.504114 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:02.504094 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce\": container with ID starting with 5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce not found: ID does not exist" containerID="5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce" Oct 30 22:00:02.504159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.504122 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce"} err="failed to get container status \"5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce\": rpc error: code = NotFound desc = could not find container \"5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce\": container with ID starting with 5018ec509c9d013ded3b130b88078e812f309aee3a0237cda69e1ad3fe8dd2ce not found: ID does not exist" Oct 30 22:00:02.654773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.654683 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx"] Oct 30 22:00:02.660842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:02.660812 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4705e99c-7ff987c44b-jb2zx"] Oct 30 22:00:03.874709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:03.874678 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519f6640-0db9-43ac-a221-fd8c2def0a81" path="/var/lib/kubelet/pods/519f6640-0db9-43ac-a221-fd8c2def0a81/volumes" Oct 30 22:00:04.274512 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:04.274481 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q"] Oct 30 22:00:04.274694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:04.274673 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" containerID="cri-o://c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517" gracePeriod=30 Oct 30 22:00:05.706647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.706609 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp"] Oct 30 22:00:05.707135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.707125 4991 cpu_manager.go:414] "containerMap: removing container" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" Oct 30 22:00:05.707207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.707140 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbd8d30-28e1-44d4-880c-25bcd3e67696" containerName="istio-proxy" Oct 30 22:00:05.707207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.707149 4991 cpu_manager.go:414] "containerMap: removing container" podUID="519f6640-0db9-43ac-a221-fd8c2def0a81" containerName="maas-api" Oct 30 22:00:05.707207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.707155 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519f6640-0db9-43ac-a221-fd8c2def0a81" containerName="maas-api" Oct 30 22:00:05.707302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.707250 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="519f6640-0db9-43ac-a221-fd8c2def0a81" containerName="maas-api" Oct 30 22:00:05.710933 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.710913 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.766808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.766758 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e65019-openshift-default-dockercfg-jjq2k\"" Oct 30 22:00:05.817753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.817717 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp"] Oct 30 22:00:05.837713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837677 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.837900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837734 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.837900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837761 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.837900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837791 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.837900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837854 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.838112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837904 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.838112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837936 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.838112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.837961 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knj2n\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-kube-api-access-knj2n\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.838112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.838081 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939168 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939440 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939219 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939440 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939242 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939440 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939261 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knj2n\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-kube-api-access-knj2n\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939440 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939391 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939440 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939432 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939464 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939488 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939515 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.939880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.939777 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.940045 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.940022 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.940150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.940098 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.940150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.940135 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.940244 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.940175 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.941692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.941665 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.942093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.942076 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.997635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.997596 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:05.997807 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:05.997782 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knj2n\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-kube-api-access-knj2n\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:06.024420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:06.024390 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:06.453580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:06.453501 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp"] Oct 30 22:00:06.458388 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:00:06.458360 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8157601_ca44_4ce8_9a43_1bcf4f7dfa64.slice/crio-439bf4d6f9f95faae68aba8f1ae85d4c6d0afb2cd4567e8fc567ae83b8cd2546 WatchSource:0}: Error finding container 439bf4d6f9f95faae68aba8f1ae85d4c6d0afb2cd4567e8fc567ae83b8cd2546: Status 404 returned error can't find the container with id 439bf4d6f9f95faae68aba8f1ae85d4c6d0afb2cd4567e8fc567ae83b8cd2546 Oct 30 22:00:06.460628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:06.460601 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:06.460687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:06.460661 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:06.460723 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:06.460692 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:06.492142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:06.492121 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 22:00:06.519536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:06.519506 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" event={"ID":"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64","Type":"ContainerStarted","Data":"439bf4d6f9f95faae68aba8f1ae85d4c6d0afb2cd4567e8fc567ae83b8cd2546"} Oct 30 22:00:07.526651 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:07.526616 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" event={"ID":"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64","Type":"ContainerStarted","Data":"7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4"} Oct 30 22:00:07.691698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:07.691643 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podStartSLOduration=2.691629015 podStartE2EDuration="2.691629015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:07.689944087 +0000 UTC m=+827.989353968" watchObservedRunningTime="2026-10-30 22:00:07.691629015 +0000 UTC m=+827.991038957" Oct 30 22:00:08.024671 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:08.024637 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:08.029542 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:08.029519 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:08.470428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:08.470347 4991 patch_prober.go:29] interesting pod/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:00:08.470428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:08.470401 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:00:08.532746 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:08.532719 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:08.533548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:08.533528 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:09.532155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.532132 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 22:00:09.539332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.539254 4991 generic.go:356] "Generic (PLEG): container finished" podID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerID="c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517" exitCode=0 Oct 30 22:00:09.539332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.539293 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" Oct 30 22:00:09.539332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.539329 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" event={"ID":"7bf81202-ac7f-478e-9819-2e3030e9bd78","Type":"ContainerDied","Data":"c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517"} Oct 30 22:00:09.539746 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.539398 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q" event={"ID":"7bf81202-ac7f-478e-9819-2e3030e9bd78","Type":"ContainerDied","Data":"75f4053a55b640014a80cb0707ede0ebcdd0c194c3502e41173f2ecfe69910ee"} Oct 30 22:00:09.539746 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.539420 4991 scope.go:122] "RemoveContainer" containerID="c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517" Oct 30 22:00:09.548878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.548861 4991 scope.go:122] "RemoveContainer" containerID="4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c" Oct 30 22:00:09.557140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.557123 4991 scope.go:122] "RemoveContainer" containerID="c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517" Oct 30 22:00:09.557455 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:09.557427 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517\": container with ID starting with c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517 not found: ID does not exist" containerID="c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517" Oct 30 22:00:09.557537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.557465 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517"} err="failed to get container status \"c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517\": rpc error: code = NotFound desc = could not find container \"c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517\": container with ID starting with c80f153ea171cad2ef2d7f8ea0cb83bf65eabf9527ebca8e03ee308f6641d517 not found: ID does not exist" Oct 30 22:00:09.557537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.557484 4991 scope.go:122] "RemoveContainer" containerID="4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c" Oct 30 22:00:09.558001 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:09.557977 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c\": container with ID starting with 4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c not found: ID does not exist" containerID="4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c" Oct 30 22:00:09.558070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.558007 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c"} err="failed to get container status \"4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c\": rpc error: code = NotFound desc = could not find container \"4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c\": container with ID starting with 4a4aa98de3513f1973ed444186e5b799a2313c72905e5115c45bb148adfcce2c not found: ID does not exist" Oct 30 22:00:09.690595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690561 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf81202-ac7f-478e-9819-2e3030e9bd78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf81202-ac7f-478e-9819-2e3030e9bd78-istiod-ca-cert\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690614 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-envoy\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690639 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-credential-socket\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690698 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-certs\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690739 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-data\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690765 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-socket\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690824 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-podinfo\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690878 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-token\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-token\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.690977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.690913 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-kube-api-access-hvc9x\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-kube-api-access-hvc9x\") pod \"7bf81202-ac7f-478e-9819-2e3030e9bd78\" (UID: \"7bf81202-ac7f-478e-9819-2e3030e9bd78\") " Oct 30 22:00:09.691215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691174 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-certs" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:09.691215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691200 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-data" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:09.691412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691258 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-credential-socket" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:09.691473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691444 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-socket" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:09.691527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691467 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf81202-ac7f-478e-9819-2e3030e9bd78-istiod-ca-cert" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:09.691609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691588 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.691724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691614 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.691724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691629 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.691724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.691644 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.693468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.693441 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-envoy" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:09.693468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.693453 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-podinfo" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:09.693591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.693469 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-kube-api-access-hvc9x" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "kube-api-access-hvc9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:09.693591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.693551 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-token" pod "7bf81202-ac7f-478e-9819-2e3030e9bd78" (UID: "7bf81202-ac7f-478e-9819-2e3030e9bd78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:09.792913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.792854 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.792913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.792878 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvc9x\" (UniqueName: \"kubernetes.io/projected/7bf81202-ac7f-478e-9819-2e3030e9bd78-kube-api-access-hvc9x\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.792913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.792889 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf81202-ac7f-478e-9819-2e3030e9bd78-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.792913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.792898 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.792913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.792907 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf81202-ac7f-478e-9819-2e3030e9bd78-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.971407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.971379 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q"] Oct 30 22:00:09.992213 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:09.992179 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4705e99c-openshift-default-54c9fb7bb5-ghn7q"] Oct 30 22:00:11.874336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:11.874299 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" path="/var/lib/kubelet/pods/7bf81202-ac7f-478e-9819-2e3030e9bd78/volumes" Oct 30 22:00:17.933463 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:17.933417 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:00:18.887568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:18.887533 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75"] Oct 30 22:00:18.888045 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:18.888029 4991 cpu_manager.go:414] "containerMap: removing container" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" Oct 30 22:00:18.888045 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:18.888045 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" Oct 30 22:00:18.888168 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:18.888153 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" Oct 30 22:00:18.893083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:18.893065 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:18.946331 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:18.946298 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-e524ce-serving-cert\"" Oct 30 22:00:18.995648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:18.995616 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75"] Oct 30 22:00:19.081178 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.081146 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d4fd17-1756-47a6-81df-e3d7174555f0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.081312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.081225 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6jr\" (UniqueName: \"kubernetes.io/projected/92d4fd17-1756-47a6-81df-e3d7174555f0-kube-api-access-nb6jr\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.081386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.081365 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-metrics-certs\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.081446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.081398 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.182342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.182230 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.182513 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.182404 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d4fd17-1756-47a6-81df-e3d7174555f0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.182513 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:19.182420 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-serving-cert: secret "maas-api-e2e-worker-w4-e524ce-serving-cert" not found Oct 30 22:00:19.182513 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:19.182502 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls podName:92d4fd17-1756-47a6-81df-e3d7174555f0 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:19.682482499 +0000 UTC m=+839.981892360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls") pod "maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" (UID: "92d4fd17-1756-47a6-81df-e3d7174555f0") : secret "maas-api-e2e-worker-w4-e524ce-serving-cert" not found Oct 30 22:00:19.182699 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.182577 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6jr\" (UniqueName: \"kubernetes.io/projected/92d4fd17-1756-47a6-81df-e3d7174555f0-kube-api-access-nb6jr\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.182699 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.182684 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-metrics-certs\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.183166 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.183145 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d4fd17-1756-47a6-81df-e3d7174555f0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.185025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.185007 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-metrics-certs\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.244299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.244270 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6jr\" (UniqueName: \"kubernetes.io/projected/92d4fd17-1756-47a6-81df-e3d7174555f0-kube-api-access-nb6jr\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.686974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.686935 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.689368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.689309 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:19.806500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:19.806462 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:20.185426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:20.185393 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75"] Oct 30 22:00:20.191240 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:00:20.191210 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d4fd17_1756_47a6_81df_e3d7174555f0.slice/crio-4c94502d9c91dc3233539b680c3de3c42a20f52eefafe194fcbeed4e74a9c66c WatchSource:0}: Error finding container 4c94502d9c91dc3233539b680c3de3c42a20f52eefafe194fcbeed4e74a9c66c: Status 404 returned error can't find the container with id 4c94502d9c91dc3233539b680c3de3c42a20f52eefafe194fcbeed4e74a9c66c Oct 30 22:00:20.621401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:20.621370 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" event={"ID":"92d4fd17-1756-47a6-81df-e3d7174555f0","Type":"ContainerStarted","Data":"4c94502d9c91dc3233539b680c3de3c42a20f52eefafe194fcbeed4e74a9c66c"} Oct 30 22:00:21.630274 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:21.630218 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" event={"ID":"92d4fd17-1756-47a6-81df-e3d7174555f0","Type":"ContainerStarted","Data":"cf79fbdd97b89a0248e677dfa4cf7352cc282eb5a8d04fc8ebddaba404371a35"} Oct 30 22:00:21.630823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:21.630333 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:21.748507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:21.748454 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" podStartSLOduration=2.895155354 podStartE2EDuration="3.748440199s" totalImagesPullingTime="853.284845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:21.748004368 +0000 UTC m=+842.047414268" watchObservedRunningTime="2026-10-30 22:00:21.748440199 +0000 UTC m=+842.047850078" Oct 30 22:00:27.642090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:27.642060 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:00:34.201238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.201204 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5"] Oct 30 22:00:34.201934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.201911 4991 cpu_manager.go:414] "containerMap: removing container" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" Oct 30 22:00:34.202036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.201935 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" Oct 30 22:00:34.202093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.202061 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf81202-ac7f-478e-9819-2e3030e9bd78" containerName="istio-proxy" Oct 30 22:00:34.205162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.205139 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.300833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.300797 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e65019-serving-cert\"" Oct 30 22:00:34.303540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.303512 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5"] Oct 30 22:00:34.355538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.355497 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/895dfcd2-d9dd-4120-98e9-12f581eee595-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.355742 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.355683 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-metrics-certs\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.355812 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.355770 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmch\" (UniqueName: \"kubernetes.io/projected/895dfcd2-d9dd-4120-98e9-12f581eee595-kube-api-access-vfmch\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.355927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.355835 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.456736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.456643 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/895dfcd2-d9dd-4120-98e9-12f581eee595-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.456920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.456755 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-metrics-certs\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.456920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.456796 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmch\" (UniqueName: \"kubernetes.io/projected/895dfcd2-d9dd-4120-98e9-12f581eee595-kube-api-access-vfmch\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.456920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.456858 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.457108 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:34.456984 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-serving-cert: secret "maas-api-e2e-worker-w1-e65019-serving-cert" not found Oct 30 22:00:34.457108 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:34.457052 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls podName:895dfcd2-d9dd-4120-98e9-12f581eee595 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:34.957033252 +0000 UTC m=+855.256443129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls") pod "maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" (UID: "895dfcd2-d9dd-4120-98e9-12f581eee595") : secret "maas-api-e2e-worker-w1-e65019-serving-cert" not found Oct 30 22:00:34.457499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.457474 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/895dfcd2-d9dd-4120-98e9-12f581eee595-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.459486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.459462 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-metrics-certs\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.515406 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.515376 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmch\" (UniqueName: \"kubernetes.io/projected/895dfcd2-d9dd-4120-98e9-12f581eee595-kube-api-access-vfmch\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.962747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.962703 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:34.965217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:34.965190 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:35.140362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:35.140328 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:35.556255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:35.556160 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5"] Oct 30 22:00:35.562000 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:00:35.561958 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895dfcd2_d9dd_4120_98e9_12f581eee595.slice/crio-9c0bc4b50ba55319426f342b32b7497f81e0327eb3d3b59215e92a478e487207 WatchSource:0}: Error finding container 9c0bc4b50ba55319426f342b32b7497f81e0327eb3d3b59215e92a478e487207: Status 404 returned error can't find the container with id 9c0bc4b50ba55319426f342b32b7497f81e0327eb3d3b59215e92a478e487207 Oct 30 22:00:35.718518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:35.718480 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" event={"ID":"895dfcd2-d9dd-4120-98e9-12f581eee595","Type":"ContainerStarted","Data":"9c0bc4b50ba55319426f342b32b7497f81e0327eb3d3b59215e92a478e487207"} Oct 30 22:00:36.726743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:36.726698 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" event={"ID":"895dfcd2-d9dd-4120-98e9-12f581eee595","Type":"ContainerStarted","Data":"d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524"} Oct 30 22:00:36.727137 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:36.726807 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:36.841581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:36.841474 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" podStartSLOduration=1.880170239 podStartE2EDuration="2.841455647s" totalImagesPullingTime="961.285408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:36.840694623 +0000 UTC m=+857.140104522" watchObservedRunningTime="2026-10-30 22:00:36.841455647 +0000 UTC m=+857.140865531" Oct 30 22:00:37.741310 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:37.741277 4991 generic.go:356] "Generic (PLEG): container finished" podID="13dca42e-3bfb-4105-a04b-f5c813540911" containerID="fd2fdf38e1a0d9e52f76e677a6890c578303f5f25385c4b4f96dc9122ac805bc" exitCode=0 Oct 30 22:00:37.741765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:37.741354 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" event={"ID":"13dca42e-3bfb-4105-a04b-f5c813540911","Type":"ContainerDied","Data":"fd2fdf38e1a0d9e52f76e677a6890c578303f5f25385c4b4f96dc9122ac805bc"} Oct 30 22:00:37.741841 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:37.741776 4991 scope.go:122] "RemoveContainer" containerID="fd2fdf38e1a0d9e52f76e677a6890c578303f5f25385c4b4f96dc9122ac805bc" Oct 30 22:00:37.742905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:37.742879 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:37.742974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:37.742939 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:37.743013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:37.742976 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:38.752405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:38.752367 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" event={"ID":"13dca42e-3bfb-4105-a04b-f5c813540911","Type":"ContainerStarted","Data":"a2b111d251ec41c98a8266345db931f60449ff5a12cca21cde1d61743a151c4b"} Oct 30 22:00:39.441443 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:39.441409 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 22:00:39.595970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:39.595940 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 22:00:39.597225 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:39.597198 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:00:39.597407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:39.597252 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 22:00:40.595859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:40.595825 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:00:40.596247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:40.595878 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 22:00:41.595448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:41.595411 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:00:41.595630 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:41.595475 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 22:00:42.595796 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:42.595762 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:00:42.596204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:42.595814 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 22:00:42.748623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:42.748593 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:00:43.595301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:43.595266 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:00:43.595487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:43.595329 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 22:00:44.599244 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:44.599208 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 22:00:44.600263 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:44.600240 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 22:00:47.911060 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:00:47.911018 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:00:54.286713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.286675 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": read tcp 10.132.0.2:57196->10.132.0.72:15021: read: connection reset by peer" start-of-body= Oct 30 22:00:54.287205 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.286745 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": read tcp 10.132.0.2:57196->10.132.0.72:15021: read: connection reset by peer" Oct 30 22:00:54.883523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.883487 4991 generic.go:356] "Generic (PLEG): container finished" podID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerID="7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4" exitCode=0 Oct 30 22:00:54.883682 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.883560 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" event={"ID":"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64","Type":"ContainerDied","Data":"7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4"} Oct 30 22:00:54.883884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.883867 4991 scope.go:122] "RemoveContainer" containerID="7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4" Oct 30 22:00:54.885037 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.885013 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:54.885087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.885075 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:54.885125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:54.885107 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:55.890280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:55.890240 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" event={"ID":"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64","Type":"ContainerStarted","Data":"7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8"} Oct 30 22:00:56.031707 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:56.031674 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:00:56.032044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:56.032024 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:00:56.032184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:56.032162 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 22:00:57.025691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:57.025654 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:00:57.026262 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:57.026231 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 22:00:58.032343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:58.027637 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:00:58.032343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:58.027695 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 22:00:59.029852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:59.029813 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:00:59.030175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:00:59.030126 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 22:01:00.035342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:00.031684 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:01:00.035342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:00.031746 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 22:01:01.029252 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:01.029217 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:01:01.029628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:01.029597 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:01:01.030245 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:01.030226 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:01:06.717906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.717871 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5"] Oct 30 22:01:06.721839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.721810 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768008 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.767980 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768120 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768056 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768131 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768228 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768201 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768284 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768255 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6lzb\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-kube-api-access-b6lzb\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768381 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768434 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768532 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.768584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.768564 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.819815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.819788 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5"] Oct 30 22:01:06.869598 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869572 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.869745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869614 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.869745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869669 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.869745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869685 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6lzb\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-kube-api-access-b6lzb\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.869745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869738 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869771 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869948 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.869985 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.870018 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870249 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.870031 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870249 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.870015 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.870273 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.870423 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.870644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.870619 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.872477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.872452 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.872724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.872702 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.928118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.928067 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:06.929017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:06.928993 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6lzb\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-kube-api-access-b6lzb\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:07.035625 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:07.035573 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:07.474495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:07.474447 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5"] Oct 30 22:01:07.478306 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:01:07.478275 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12bfbdc_ad52_4cc3_9f8e_8b68603d66eb.slice/crio-5a557b8e4f40400ed75104a21a51c40cdfad38ca77dcd5abd9d970e284859a95 WatchSource:0}: Error finding container 5a557b8e4f40400ed75104a21a51c40cdfad38ca77dcd5abd9d970e284859a95: Status 404 returned error can't find the container with id 5a557b8e4f40400ed75104a21a51c40cdfad38ca77dcd5abd9d970e284859a95 Oct 30 22:01:07.480692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:07.480666 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:01:07.480762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:07.480722 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:01:07.480762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:07.480748 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:01:08.006519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:08.006483 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" event={"ID":"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb","Type":"ContainerStarted","Data":"1e127c34f23913fe20034c10cb02363b8db2557cce83feef610869f6bb3b586c"} Oct 30 22:01:08.006975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:08.006562 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" event={"ID":"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb","Type":"ContainerStarted","Data":"5a557b8e4f40400ed75104a21a51c40cdfad38ca77dcd5abd9d970e284859a95"} Oct 30 22:01:08.036708 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:08.036676 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:08.038191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:08.038165 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:01:08.038305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:08.038208 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 22:01:09.044511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:09.044464 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:01:09.044999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:09.044525 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 22:01:10.042710 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:10.042676 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:01:10.042954 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:10.042926 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 22:01:11.042342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:11.041661 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:01:11.042342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:11.041757 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 22:01:12.044464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:12.044423 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:01:12.044973 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:12.044493 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 22:01:13.043447 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:13.043409 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:01:13.043732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:13.043705 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 22:01:14.040658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:14.040626 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:14.041042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:14.041007 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:14.041724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:14.041703 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:01:14.213670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:14.213622 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podStartSLOduration=8.213609224 podStartE2EDuration="8.213609224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.128935077 +0000 UTC m=+888.428344959" watchObservedRunningTime="2026-10-30 22:01:14.213609224 +0000 UTC m=+894.513019104" Oct 30 22:01:17.910192 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:01:17.910158 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:01:20.005051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:20.005023 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:01:20.009202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:01:20.009180 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:01:47.946622 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:01:47.946581 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:02:17.910512 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:02:17.910468 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:02:31.085986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.085945 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54"] Oct 30 22:02:31.090911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.090883 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.142942 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.142911 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b47e8d-openshift-default-dockercfg-bxx6t\"" Oct 30 22:02:31.192527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.192491 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54"] Oct 30 22:02:31.248367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248315 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248399 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248425 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbp5\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-kube-api-access-fvbp5\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248556 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248582 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248671 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248604 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248763 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248829 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.248883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.248871 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359372 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359434 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359471 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359504 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359637 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359664 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359691 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbp5\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-kube-api-access-fvbp5\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359783 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.359810 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.360559 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.360787 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.361002 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.361610 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.363389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.362219 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.386353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.384866 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.403354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.403304 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.441375 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.439762 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.441890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.441851 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbp5\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-kube-api-access-fvbp5\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:31.712372 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:31.712267 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:32.289518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:32.289484 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54"] Oct 30 22:02:32.297572 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:02:32.297544 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54adce1e_7ecb_40cb_9d22_8ee7fde39dc1.slice/crio-dbdbe77a50fd47ac9d56df7e4578da1620e899a511dd1c4586dd64633025950e WatchSource:0}: Error finding container dbdbe77a50fd47ac9d56df7e4578da1620e899a511dd1c4586dd64633025950e: Status 404 returned error can't find the container with id dbdbe77a50fd47ac9d56df7e4578da1620e899a511dd1c4586dd64633025950e Oct 30 22:02:32.300389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:32.300357 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:32.300499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:32.300446 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:32.300499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:32.300490 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:32.585858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:32.585767 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" event={"ID":"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1","Type":"ContainerStarted","Data":"81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526"} Oct 30 22:02:32.585858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:32.585844 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" event={"ID":"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1","Type":"ContainerStarted","Data":"dbdbe77a50fd47ac9d56df7e4578da1620e899a511dd1c4586dd64633025950e"} Oct 30 22:02:32.712498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:32.712460 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:33.714745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:33.714700 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:33.715128 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:33.714780 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 22:02:34.713330 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:34.713290 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:34.713505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:34.713370 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 22:02:35.713149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:35.713111 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:35.713642 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:35.713165 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 22:02:36.713222 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:36.713187 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:36.713798 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:36.713244 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 22:02:37.712691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:37.712658 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:37.712899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:37.712721 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 22:02:38.713281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:38.713236 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:38.713764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:38.713314 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 22:02:39.717415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:39.717385 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:39.717825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:39.717804 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:39.718571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:39.718549 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:02:39.906201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:39.906149 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podStartSLOduration=8.906136288 podStartE2EDuration="8.906136288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:32.711584817 +0000 UTC m=+973.010994696" watchObservedRunningTime="2026-10-30 22:02:39.906136288 +0000 UTC m=+980.205546168" Oct 30 22:02:47.910788 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:02:47.910742 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:02:53.350136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.350100 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc"] Oct 30 22:02:53.354551 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.354527 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.405905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.405882 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7f66ab-openshift-default-dockercfg-pk2pq\"" Oct 30 22:02:53.453838 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.453813 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc"] Oct 30 22:02:53.459145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459118 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459179 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d2e12e-be2c-4aea-bf06-4643e567b804-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459271 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459360 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459303 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459415 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459436 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459461 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459481 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.459628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.459606 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngjp\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-kube-api-access-cngjp\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561176 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561213 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561244 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561260 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561286 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561305 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561375 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngjp\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-kube-api-access-cngjp\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561439 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561484 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d2e12e-be2c-4aea-bf06-4643e567b804-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561683 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561754 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.561952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561784 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.562012 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.561963 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.562147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.562122 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d2e12e-be2c-4aea-bf06-4643e567b804-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.564255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.564227 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.564483 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.564464 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.619065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.619011 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.619191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.619172 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngjp\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-kube-api-access-cngjp\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:53.667949 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:53.667924 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:54.101794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:54.101757 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc"] Oct 30 22:02:54.107425 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:02:54.107394 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d2e12e_be2c_4aea_bf06_4643e567b804.slice/crio-ddf5027380db737f36b9bfef2a70f1ee7bffa32bd582e8050a395eda693c9b66 WatchSource:0}: Error finding container ddf5027380db737f36b9bfef2a70f1ee7bffa32bd582e8050a395eda693c9b66: Status 404 returned error can't find the container with id ddf5027380db737f36b9bfef2a70f1ee7bffa32bd582e8050a395eda693c9b66 Oct 30 22:02:54.109563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:54.109531 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:54.109627 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:54.109596 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:54.109668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:54.109625 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:54.731905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:54.731858 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" event={"ID":"80d2e12e-be2c-4aea-bf06-4643e567b804","Type":"ContainerStarted","Data":"e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380"} Oct 30 22:02:54.732526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:54.732499 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" event={"ID":"80d2e12e-be2c-4aea-bf06-4643e567b804","Type":"ContainerStarted","Data":"ddf5027380db737f36b9bfef2a70f1ee7bffa32bd582e8050a395eda693c9b66"} Oct 30 22:02:54.849459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:54.849405 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podStartSLOduration=1.84938833 podStartE2EDuration="1.84938833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:54.846958104 +0000 UTC m=+995.146368005" watchObservedRunningTime="2026-10-30 22:02:54.84938833 +0000 UTC m=+995.148798209" Oct 30 22:02:55.668436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:55.668283 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:02:55.672344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:55.670297 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:02:55.672344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:55.670378 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 22:02:56.668780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:56.668733 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:02:56.669296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:56.668799 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 22:02:57.668309 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:57.668272 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:02:57.668502 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:57.668369 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 22:02:58.669112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:58.669067 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:02:58.669630 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:58.669126 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 22:02:59.669084 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:59.669053 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:02:59.669355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:02:59.669303 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 22:03:01.477550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:01.477490 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:03:01.478037 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:01.477961 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:03:01.478820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:01.478798 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:03:17.910642 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:03:17.910607 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:03:47.910752 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:03:47.910715 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:03:53.821583 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.821544 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd"] Oct 30 22:03:53.825837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.825818 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872375 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872697 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872781 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872825 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872858 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872884 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872910 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.872958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872946 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.873164 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.872996 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4hb\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-kube-api-access-bs4hb\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.909910 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.909878 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd"] Oct 30 22:03:53.977507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.977467 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.978214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.978188 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.978475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.978454 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.978644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.978627 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.978768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.978752 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.978876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.978862 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.978979 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.978963 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.980497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.979055 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.980497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.979078 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.980497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.979391 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.980497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.979620 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.980497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.979758 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.980497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.979850 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.980497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.980123 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4hb\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-kube-api-access-bs4hb\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.991721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.991659 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:53.999499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:53.999442 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:54.037850 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:54.037809 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:54.038007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:54.037911 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4hb\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-kube-api-access-bs4hb\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:54.139277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:54.139204 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:54.574855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:54.574818 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd"] Oct 30 22:03:54.581831 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:03:54.581798 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a555122_264a_4159_a3a5_e2e9cb62c5aa.slice/crio-592bb5c3c7b2a1182d0e9d6652061d7180b3ea7ec2f0a0d763b1c96b3dc00b45 WatchSource:0}: Error finding container 592bb5c3c7b2a1182d0e9d6652061d7180b3ea7ec2f0a0d763b1c96b3dc00b45: Status 404 returned error can't find the container with id 592bb5c3c7b2a1182d0e9d6652061d7180b3ea7ec2f0a0d763b1c96b3dc00b45 Oct 30 22:03:54.586065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:54.586024 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:54.586221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:54.586115 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:54.586221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:54.586154 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:55.164140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:55.164107 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" event={"ID":"1a555122-264a-4159-a3a5-e2e9cb62c5aa","Type":"ContainerStarted","Data":"37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192"} Oct 30 22:03:55.164552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:55.164156 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" event={"ID":"1a555122-264a-4159-a3a5-e2e9cb62c5aa","Type":"ContainerStarted","Data":"592bb5c3c7b2a1182d0e9d6652061d7180b3ea7ec2f0a0d763b1c96b3dc00b45"} Oct 30 22:03:55.319338 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:55.318187 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podStartSLOduration=2.318169608 podStartE2EDuration="2.318169608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:55.317086443 +0000 UTC m=+1055.616496323" watchObservedRunningTime="2026-10-30 22:03:55.318169608 +0000 UTC m=+1055.617579488" Oct 30 22:03:56.142343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:56.141641 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:03:56.145398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:56.143917 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:03:56.145398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:56.143974 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:03:57.141735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:57.141693 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:03:57.142362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:57.142305 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:03:58.142724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.142687 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:03:58.143296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.143265 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:03:58.199164 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.199133 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6"] Oct 30 22:03:58.203352 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.203331 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.357164 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6"] Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360507 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360727 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360760 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnnt\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-kube-api-access-bxnnt\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360789 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360829 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360884 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360967 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.360997 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b68edd-8617-43ae-9430-6616068a1698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.364813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.361054 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b68edd-8617-43ae-9430-6616068a1698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462616 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462684 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462703 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnnt\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-kube-api-access-bxnnt\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462730 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462758 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462800 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462855 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462883 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b68edd-8617-43ae-9430-6616068a1698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.462918 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b68edd-8617-43ae-9430-6616068a1698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.463601 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b68edd-8617-43ae-9430-6616068a1698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.464498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.463817 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.465218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.464640 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.465218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.464950 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.465218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.465194 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.468120 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.468090 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b68edd-8617-43ae-9430-6616068a1698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.468352 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.468313 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.552412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.552381 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.553083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.553057 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnnt\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-kube-api-access-bxnnt\") pod \"maas-default-gateway-openshift-default-6f54779779-fpmc6\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:58.817568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:58.817536 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:03:59.141137 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:59.140630 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:03:59.141137 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:59.140695 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:03:59.275750 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:59.275720 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6"] Oct 30 22:03:59.280683 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:03:59.280653 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b68edd_8617_43ae_9430_6616068a1698.slice/crio-cc597560bb0c838c250b181dff57d49929c461f35450e2329300710f3231f2b4 WatchSource:0}: Error finding container cc597560bb0c838c250b181dff57d49929c461f35450e2329300710f3231f2b4: Status 404 returned error can't find the container with id cc597560bb0c838c250b181dff57d49929c461f35450e2329300710f3231f2b4 Oct 30 22:03:59.283149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:59.283117 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:59.283243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:59.283178 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:59.283243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:03:59.283207 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:04:00.141346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.140741 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:04:00.141346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.140801 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:04:00.196202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.196163 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" event={"ID":"b2b68edd-8617-43ae-9430-6616068a1698","Type":"ContainerStarted","Data":"f21965bc286bf80fe5acbe19e4e94df7a72fbe48344788ab35466ae231554fca"} Oct 30 22:04:00.196447 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.196219 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" event={"ID":"b2b68edd-8617-43ae-9430-6616068a1698","Type":"ContainerStarted","Data":"cc597560bb0c838c250b181dff57d49929c461f35450e2329300710f3231f2b4"} Oct 30 22:04:00.430402 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.429643 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podStartSLOduration=2.429623088 podStartE2EDuration="2.429623088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:00.377785783 +0000 UTC m=+1060.677195688" watchObservedRunningTime="2026-10-30 22:04:00.429623088 +0000 UTC m=+1060.729032969" Oct 30 22:04:00.817774 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.817733 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:04:00.819210 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.819174 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:04:00.819388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:00.819238 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 22:04:01.818546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:01.818502 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:04:01.819150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:01.819118 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 22:04:02.194580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:02.194472 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:02.194875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:02.194826 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:02.818769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:02.818726 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:04:02.819399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:02.819362 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 22:04:02.918223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:02.918190 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:04:02.919031 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:02.918956 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:04:02.925587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:02.925526 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:04:03.818787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:03.818749 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:04:03.819433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:03.819398 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 22:04:04.818553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:04.818519 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:04:04.818707 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:04.818568 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 22:04:05.818868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:05.818832 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:04:05.819259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:05.818895 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 22:04:06.818461 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:06.818414 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:04:06.818666 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:06.818480 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 22:04:08.819237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:08.819200 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:08.819779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:08.819257 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:09.860504 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:09.860461 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:09.861025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:09.860540 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:10.472922 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:10.472888 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:04:10.473372 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:10.473351 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:04:10.474194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:10.474177 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:04:13.294757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.294720 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn"] Oct 30 22:04:13.298967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.298941 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.400509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.400471 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.400509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.400510 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.400756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.400542 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhp4b\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-kube-api-access-bhp4b\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.400756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.400691 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.400876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.400812 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.400932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.400919 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.400986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.400957 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.401083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.401064 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.401146 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.401088 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f77b228e-1689-4765-bd71-1e55d0e3c498-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.496731 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.496700 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn"] Oct 30 22:04:13.502500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502467 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502520 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502579 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502603 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f77b228e-1689-4765-bd71-1e55d0e3c498-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502654 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502898 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502688 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502898 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502735 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhp4b\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-kube-api-access-bhp4b\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502898 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502796 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.502898 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.502862 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.503574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.503547 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.503997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.503964 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f77b228e-1689-4765-bd71-1e55d0e3c498-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.504109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.504050 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.504295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.504250 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.504458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.504431 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.506094 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.505663 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.506672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.506650 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.563151 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.563093 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.563259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.563229 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhp4b\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-kube-api-access-bhp4b\") pod \"maas-default-gateway-openshift-default-6f54779779-x2fbn\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:13.614502 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:13.614480 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:14.045686 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.045658 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn"] Oct 30 22:04:14.051012 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:04:14.050986 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77b228e_1689_4765_bd71_1e55d0e3c498.slice/crio-38d103e578318c5e4ab4d36d530e316a08ae26335f44a72c8b5f54d04345517e WatchSource:0}: Error finding container 38d103e578318c5e4ab4d36d530e316a08ae26335f44a72c8b5f54d04345517e: Status 404 returned error can't find the container with id 38d103e578318c5e4ab4d36d530e316a08ae26335f44a72c8b5f54d04345517e Oct 30 22:04:14.053146 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.053115 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:04:14.053229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.053180 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:04:14.053229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.053206 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:04:14.421846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.421763 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" event={"ID":"f77b228e-1689-4765-bd71-1e55d0e3c498","Type":"ContainerStarted","Data":"8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7"} Oct 30 22:04:14.422282 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.421853 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" event={"ID":"f77b228e-1689-4765-bd71-1e55d0e3c498","Type":"ContainerStarted","Data":"38d103e578318c5e4ab4d36d530e316a08ae26335f44a72c8b5f54d04345517e"} Oct 30 22:04:14.537838 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.537787 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podStartSLOduration=1.5377730440000001 podStartE2EDuration="1.537773044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:14.53548885 +0000 UTC m=+1074.834898750" watchObservedRunningTime="2026-10-30 22:04:14.537773044 +0000 UTC m=+1074.837182921" Oct 30 22:04:14.615393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:14.615360 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:15.616596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:15.616558 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:04:15.616993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:15.616621 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:04:16.615687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:16.615652 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:04:16.615859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:16.615712 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:04:17.615140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:17.615102 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:04:17.615560 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:17.615166 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:04:17.910870 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:04:17.910787 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:04:18.615397 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:18.615358 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:04:18.615821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:18.615414 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:04:19.615159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:19.615116 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:04:19.615364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:19.615194 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:04:20.614962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:20.614918 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:04:20.615507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:20.614984 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:04:21.614880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:21.614849 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:04:21.615104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:21.615082 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:04:23.616586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:23.616545 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:23.617095 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:23.616596 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:24.657499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:24.657448 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:24.658083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:24.657507 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:25.387678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:25.387646 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:25.388580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:25.388549 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:25.389725 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:04:25.389705 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:04:47.916056 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:04:47.916012 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:05:08.562415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.562337 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv"] Oct 30 22:05:08.566092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.566071 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.665877 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.665839 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv"] Oct 30 22:05:08.726505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726474 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726599 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726652 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726679 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726696 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726713 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726805 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726839 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.726891 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.726873 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzw4d\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-kube-api-access-gzw4d\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828348 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828392 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828436 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828461 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828490 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828626 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828860 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828685 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828860 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828734 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzw4d\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-kube-api-access-gzw4d\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.828860 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828824 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.829025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828858 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.829025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.828933 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.829176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.829152 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.829224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.829190 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.829260 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.829228 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.830785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.830758 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.830937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.830909 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.886932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.886890 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:08.887064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:08.886967 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzw4d\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-kube-api-access-gzw4d\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:09.179213 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.179125 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:09.625349 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:05:09.625286 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8cfc39_4b24_43a7_9871_1388d96d24e0.slice/crio-cc75a15e297c96a9ddd0e67001f683e6d00b719a8d3b0db90ade32fddfae019f WatchSource:0}: Error finding container cc75a15e297c96a9ddd0e67001f683e6d00b719a8d3b0db90ade32fddfae019f: Status 404 returned error can't find the container with id cc75a15e297c96a9ddd0e67001f683e6d00b719a8d3b0db90ade32fddfae019f Oct 30 22:05:09.627709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.627681 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:09.627774 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.627748 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:09.627810 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.627778 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:09.660197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.660173 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv"] Oct 30 22:05:09.850366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.850314 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" event={"ID":"3c8cfc39-4b24-43a7-9871-1388d96d24e0","Type":"ContainerStarted","Data":"2b61f60d0e9c97ff9b277ee9ea3a45894db15e75c5c9fcc6880f7e3f0c3a6e1c"} Oct 30 22:05:09.850534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.850386 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" event={"ID":"3c8cfc39-4b24-43a7-9871-1388d96d24e0","Type":"ContainerStarted","Data":"cc75a15e297c96a9ddd0e67001f683e6d00b719a8d3b0db90ade32fddfae019f"} Oct 30 22:05:09.973092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:09.972968 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podStartSLOduration=1.972953274 podStartE2EDuration="1.972953274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:09.969072406 +0000 UTC m=+1130.268482287" watchObservedRunningTime="2026-10-30 22:05:09.972953274 +0000 UTC m=+1130.272363153" Oct 30 22:05:10.180148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:10.180113 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:10.181705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:10.181681 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:10.181794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:10.181726 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:05:11.179864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:11.179820 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:11.180269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:11.179888 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:05:12.180587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:12.180493 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:12.180587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:12.180559 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:05:13.179514 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:13.179482 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:13.179683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:13.179531 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:05:14.179668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:14.179635 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:14.180060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:14.179686 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:05:15.180093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:15.180055 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:15.180481 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:15.180118 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:05:16.180642 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:16.180591 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:16.181148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:16.180674 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:05:17.183908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:17.183876 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:17.184374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:17.184352 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:17.185112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:17.185089 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:05:17.912100 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:05:17.912062 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:05:39.665302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.665269 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl"] Oct 30 22:05:39.669242 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.669225 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.706721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.706697 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.706829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.706738 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.706829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.706809 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.706917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.706844 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.706952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.706929 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.706986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.706959 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.707026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.706992 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mz5p\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-kube-api-access-9mz5p\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.707062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.707050 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.707098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.707090 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.807884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.807847 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.807931 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.807964 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808009 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808040 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808313 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808085 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mz5p\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-kube-api-access-9mz5p\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808313 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808129 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808313 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808164 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.808488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808315 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.809486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808758 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.809486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.808805 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.809486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.809436 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.810734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.810411 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.810734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.810672 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.810901 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.810857 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.812524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.812458 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.867250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.867218 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mz5p\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-kube-api-access-9mz5p\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.867490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.867473 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:39.890153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.888612 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl"] Oct 30 22:05:39.983477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:39.983452 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:40.413821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:40.413745 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl"] Oct 30 22:05:40.420093 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:05:40.420065 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9630cc_c45d_456a_b76e_84648b0cd3d4.slice/crio-f7f13a3552096986c57399749e485130472cd8b16dcb264607149caa02f16562 WatchSource:0}: Error finding container f7f13a3552096986c57399749e485130472cd8b16dcb264607149caa02f16562: Status 404 returned error can't find the container with id f7f13a3552096986c57399749e485130472cd8b16dcb264607149caa02f16562 Oct 30 22:05:40.422190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:40.422157 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:40.422276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:40.422228 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:40.422276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:40.422262 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:41.117970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:41.117931 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" event={"ID":"5b9630cc-c45d-456a-b76e-84648b0cd3d4","Type":"ContainerStarted","Data":"4cfc596da50ee1d5a570129180c32df859dca12081ca5df7c7d5045b06e86351"} Oct 30 22:05:41.118473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:41.117994 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" event={"ID":"5b9630cc-c45d-456a-b76e-84648b0cd3d4","Type":"ContainerStarted","Data":"f7f13a3552096986c57399749e485130472cd8b16dcb264607149caa02f16562"} Oct 30 22:05:41.287435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:41.287249 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podStartSLOduration=2.287230985 podStartE2EDuration="2.287230985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:41.286129709 +0000 UTC m=+1161.585539589" watchObservedRunningTime="2026-10-30 22:05:41.287230985 +0000 UTC m=+1161.586640865" Oct 30 22:05:41.984141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:41.984108 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:41.985466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:41.985439 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:41.985586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:41.985496 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 22:05:42.984723 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:42.984681 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:42.985208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:42.984751 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 22:05:43.984421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:43.984389 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:43.984592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:43.984441 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 22:05:44.984590 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:44.984554 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:44.985000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:44.984606 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 22:05:45.984142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:45.984098 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:45.984330 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:45.984168 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 22:05:46.984706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:46.984671 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:46.985087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:46.984736 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 22:05:47.910507 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:05:47.910473 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:05:47.987945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:47.987917 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:47.988435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:47.988286 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:47.989069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:47.989051 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:05:54.570088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.570039 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9"] Oct 30 22:05:54.577304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.576074 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.698080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.698048 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"default-dockercfg-dqqb4\"" Oct 30 22:05:54.701862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.698515 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"kube-root-ca.crt\"" Oct 30 22:05:54.701862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.699204 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"openshift-service-ca.crt\"" Oct 30 22:05:54.701862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.699368 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"distinct-2-w1-e65019-kserve-self-signed-certs\"" Oct 30 22:05:54.701862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.701678 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9"] Oct 30 22:05:54.709423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.707972 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-tmp-dir\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.709423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.708069 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45e44a5-3029-43ff-a870-4190e9a041ec-tls-certs\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.709423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.708119 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-model-cache\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.709423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.708177 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmn2v\" (UniqueName: \"kubernetes.io/projected/e45e44a5-3029-43ff-a870-4190e9a041ec-kube-api-access-fmn2v\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.709423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.708267 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-home\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.709423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.708366 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-dshm\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.841419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.839578 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-tmp-dir\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.841419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.839675 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45e44a5-3029-43ff-a870-4190e9a041ec-tls-certs\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.841419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.839727 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-model-cache\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.841419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.839784 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn2v\" (UniqueName: \"kubernetes.io/projected/e45e44a5-3029-43ff-a870-4190e9a041ec-kube-api-access-fmn2v\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.841419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.839869 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-home\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.841419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.839924 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-dshm\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.870376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.843719 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-model-cache\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.870376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.844002 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-tmp-dir\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.891340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.871247 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-home\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.909361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.901616 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45e44a5-3029-43ff-a870-4190e9a041ec-tls-certs\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.909361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.905813 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-dshm\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:54.983181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:54.982993 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn2v\" (UniqueName: \"kubernetes.io/projected/e45e44a5-3029-43ff-a870-4190e9a041ec-kube-api-access-fmn2v\") pod \"distinct-2-w1-e65019-kserve-76f9765b95-crck9\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:55.219396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:55.219294 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:05:55.803394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:55.803301 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9"] Oct 30 22:05:55.809705 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:05:55.809676 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode45e44a5_3029_43ff_a870_4190e9a041ec.slice/crio-6ff336ffcbd9213b75261b639eec5f30aa46631fd52afc72e5790e60a83a11f3 WatchSource:0}: Error finding container 6ff336ffcbd9213b75261b639eec5f30aa46631fd52afc72e5790e60a83a11f3: Status 404 returned error can't find the container with id 6ff336ffcbd9213b75261b639eec5f30aa46631fd52afc72e5790e60a83a11f3 Oct 30 22:05:56.303983 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:56.303948 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" event={"ID":"e45e44a5-3029-43ff-a870-4190e9a041ec","Type":"ContainerStarted","Data":"46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0"} Oct 30 22:05:56.304168 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:56.303996 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" event={"ID":"e45e44a5-3029-43ff-a870-4190e9a041ec","Type":"ContainerStarted","Data":"6ff336ffcbd9213b75261b639eec5f30aa46631fd52afc72e5790e60a83a11f3"} Oct 30 22:05:56.436139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:56.436088 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podStartSLOduration=2.436076182 podStartE2EDuration="2.436076182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:56.434533948 +0000 UTC m=+1176.733943828" watchObservedRunningTime="2026-10-30 22:05:56.436076182 +0000 UTC m=+1176.735486061" Oct 30 22:05:57.219877 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:05:57.219844 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:06:05.219786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:05.219752 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:06:05.232423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:05.232400 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:06:05.371533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:05.371502 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:06:09.053499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.053466 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n"] Oct 30 22:06:09.057436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.057408 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.163760 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.163713 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.163952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.163805 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.163952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.163833 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.163952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.163881 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.163952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.163939 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.164170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.163998 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.164170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.164022 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhgm\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-kube-api-access-jwhgm\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.164170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.164051 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87bb90a-3285-418e-920c-faf02aaf23c6-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.164170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.164136 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.172882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.172856 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n"] Oct 30 22:06:09.265283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265242 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265292 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265357 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265415 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265472 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265498 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhgm\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-kube-api-access-jwhgm\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265529 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87bb90a-3285-418e-920c-faf02aaf23c6-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265613 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.265773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.265700 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.266156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.266130 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.266437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.266412 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.266848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.266827 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.267398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.267368 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87bb90a-3285-418e-920c-faf02aaf23c6-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.267593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.267493 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.271481 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.269944 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.275917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.275893 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.328172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.328112 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.330977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.330950 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhgm\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-kube-api-access-jwhgm\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.375335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.375293 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:09.826152 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.826089 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n"] Oct 30 22:06:09.830455 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:06:09.830414 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87bb90a_3285_418e_920c_faf02aaf23c6.slice/crio-0f4e63cfc1c5a98eb6011f395d92e5cab30c8205101a4333e883e1ae5ecbf6fc WatchSource:0}: Error finding container 0f4e63cfc1c5a98eb6011f395d92e5cab30c8205101a4333e883e1ae5ecbf6fc: Status 404 returned error can't find the container with id 0f4e63cfc1c5a98eb6011f395d92e5cab30c8205101a4333e883e1ae5ecbf6fc Oct 30 22:06:09.833100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.833067 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:09.833185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.833134 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:09.833185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:09.833163 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:06:10.394301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:10.394257 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" event={"ID":"c87bb90a-3285-418e-920c-faf02aaf23c6","Type":"ContainerStarted","Data":"b9fb7da35156cfeb18ac48f506ad6d48fca082d85bec6df55f40d9b619b746e4"} Oct 30 22:06:10.394778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:10.394344 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" event={"ID":"c87bb90a-3285-418e-920c-faf02aaf23c6","Type":"ContainerStarted","Data":"0f4e63cfc1c5a98eb6011f395d92e5cab30c8205101a4333e883e1ae5ecbf6fc"} Oct 30 22:06:10.531570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:10.531503 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podStartSLOduration=2.5314835369999997 podStartE2EDuration="2.531483537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:10.526024338 +0000 UTC m=+1190.825434220" watchObservedRunningTime="2026-10-30 22:06:10.531483537 +0000 UTC m=+1190.830893413" Oct 30 22:06:11.376111 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:11.376073 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:11.377991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:11.377960 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:06:11.378138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:11.378016 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:06:12.376657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:12.376615 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:06:12.377087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:12.376678 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:06:13.376062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:13.376021 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:06:13.376243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:13.376075 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:06:14.376232 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:14.376189 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:06:14.376655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:14.376250 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:06:15.375747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:15.375711 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:06:15.375922 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:15.375773 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:06:16.375805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:16.375765 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:06:16.376199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:16.375815 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:06:17.380716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:17.380687 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:17.381187 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:17.381130 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:17.381787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:17.381767 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:06:17.911215 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:06:17.911180 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:06:20.056848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:20.056817 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:06:20.061595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:20.061575 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:06:47.911564 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:06:47.911525 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:06:50.375193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:50.375150 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:53.357330 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:53.357000 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:53.365701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:06:53.365668 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:10.414945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.414906 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc"] Oct 30 22:07:10.506345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.501609 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618296 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618379 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618412 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618518 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618548 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618570 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7bz\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-kube-api-access-gp7bz\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618603 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618725 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.666400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.618772 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724271 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724343 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724445 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724475 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724503 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7bz\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-kube-api-access-gp7bz\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724538 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724675 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724724 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.724776 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.725370 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.725652 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.753770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.725902 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.755467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.755435 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.757748 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.757725 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.902345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.877889 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.902621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.881629 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc"] Oct 30 22:07:10.957387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.914866 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.986904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.959748 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:10.986904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:10.977043 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7bz\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-kube-api-access-gp7bz\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:11.099345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:11.099269 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:12.590345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:12.575462 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:12.705410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:12.705283 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc"] Oct 30 22:07:12.711021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:12.710984 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:12.711246 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:12.711225 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:12.711435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:12.711414 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:14.136937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:14.136884 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:14.137578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:14.137524 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:07:14.172585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:14.172550 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" event={"ID":"c1bc7b36-a745-4e9f-975a-4e4cc718ee61","Type":"ContainerStarted","Data":"ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c"} Oct 30 22:07:14.172856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:14.172816 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" event={"ID":"c1bc7b36-a745-4e9f-975a-4e4cc718ee61","Type":"ContainerStarted","Data":"eee0a478e7808911175f24f7939077bd7538062ea13b1184fb27b989e0a6b8f6"} Oct 30 22:07:14.173026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:14.173011 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:15.100703 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:15.100670 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:15.100884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:15.100724 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:07:16.100139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:16.100103 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:16.100551 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:16.100165 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:07:17.100295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:17.100262 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:17.100705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:17.100313 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:07:17.911431 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:07:17.911389 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:07:18.099944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:18.099905 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:18.100159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:18.099962 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:07:19.100499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:19.100469 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:19.100987 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:19.100514 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:07:20.099911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:20.099879 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:20.100069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:20.099932 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:07:21.104846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:21.104810 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:21.105352 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:21.105294 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:21.106153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:21.106131 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:07:21.267771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:21.267708 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podStartSLOduration=11.267691889 podStartE2EDuration="11.267691889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:14.164261388 +0000 UTC m=+1254.463671270" watchObservedRunningTime="2026-10-30 22:07:21.267691889 +0000 UTC m=+1261.567101772" Oct 30 22:07:30.827714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.827674 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm"] Oct 30 22:07:30.833953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.833923 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:30.888373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.887915 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"openshift-service-ca.crt\"" Oct 30 22:07:30.898740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.891891 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"kube-root-ca.crt\"" Oct 30 22:07:30.898740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.896548 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm"] Oct 30 22:07:30.934112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.934008 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"default-dockercfg-v8p5t\"" Oct 30 22:07:30.934516 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.934494 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"distinct-w2-7f66ab-kserve-self-signed-certs\"" Oct 30 22:07:30.951100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.951043 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g47v\" (UniqueName: \"kubernetes.io/projected/6309d2d6-8637-4a0f-9520-7024b8957e5e-kube-api-access-7g47v\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:30.951290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.951180 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-tmp-dir\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:30.951290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.951239 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-model-cache\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:30.951464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.951430 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-dshm\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:30.951614 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.951596 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-home\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:30.951696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:30.951681 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6309d2d6-8637-4a0f-9520-7024b8957e5e-tls-certs\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.052616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.052569 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-model-cache\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.052797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.052716 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-dshm\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.052878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.052827 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-home\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.052878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.052876 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6309d2d6-8637-4a0f-9520-7024b8957e5e-tls-certs\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.052994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.052915 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g47v\" (UniqueName: \"kubernetes.io/projected/6309d2d6-8637-4a0f-9520-7024b8957e5e-kube-api-access-7g47v\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.052994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.052945 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-tmp-dir\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.053106 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.052995 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-model-cache\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.053233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.053207 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-home\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.053306 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.053269 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-tmp-dir\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.055494 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.055468 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-dshm\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.055619 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.055582 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6309d2d6-8637-4a0f-9520-7024b8957e5e-tls-certs\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.111425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.111348 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g47v\" (UniqueName: \"kubernetes.io/projected/6309d2d6-8637-4a0f-9520-7024b8957e5e-kube-api-access-7g47v\") pod \"distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.170876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.170599 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:31.591506 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:31.591470 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm"] Oct 30 22:07:31.597962 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:07:31.597934 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6309d2d6_8637_4a0f_9520_7024b8957e5e.slice/crio-88c40827df15997f56617972514e1036d1c494280b16053d5da29872f00fccb8 WatchSource:0}: Error finding container 88c40827df15997f56617972514e1036d1c494280b16053d5da29872f00fccb8: Status 404 returned error can't find the container with id 88c40827df15997f56617972514e1036d1c494280b16053d5da29872f00fccb8 Oct 30 22:07:32.208025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:32.207990 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" event={"ID":"6309d2d6-8637-4a0f-9520-7024b8957e5e","Type":"ContainerStarted","Data":"60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de"} Oct 30 22:07:32.208025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:32.208030 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" event={"ID":"6309d2d6-8637-4a0f-9520-7024b8957e5e","Type":"ContainerStarted","Data":"88c40827df15997f56617972514e1036d1c494280b16053d5da29872f00fccb8"} Oct 30 22:07:32.321883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:32.321833 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podStartSLOduration=2.321819497 podStartE2EDuration="2.321819497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:32.318742752 +0000 UTC m=+1272.618152652" watchObservedRunningTime="2026-10-30 22:07:32.321819497 +0000 UTC m=+1272.621229380" Oct 30 22:07:33.172049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:33.172017 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:41.172142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:41.172111 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:41.184380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:41.184342 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:41.284737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:41.284708 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:07:47.303366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:47.296435 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:47.395488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:47.395443 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:07:47.396652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:47.396624 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:47.941060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:47.941025 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd"] Oct 30 22:07:47.976797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:47.976770 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:47.979287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:47.979259 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd"] Oct 30 22:07:48.048347 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:07:48.030541 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:07:48.060003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.056649 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d09fff-openshift-default-dockercfg-596tc\"" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.301646 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.301753 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.301792 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.301827 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.301867 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdwh\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-kube-api-access-qqdwh\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.301940 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.302017 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.302041 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.305436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.302100 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403241 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403277 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdwh\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-kube-api-access-qqdwh\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403358 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403403 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403422 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403473 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403552 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403690 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403772 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403816 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.403852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403773 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.404018 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.403994 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.404250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.404228 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.404485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.404469 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.405640 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.405617 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.405946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.405929 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.461433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.461406 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.461633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.461613 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdwh\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-kube-api-access-qqdwh\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:48.672871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:48.672792 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:49.132113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.130276 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd"] Oct 30 22:07:49.140538 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:07:49.140503 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b2bf36_34d2_4ac1_8d78_2a9891ee04d7.slice/crio-200c6fb9d2f5df0841bf0bcd62ee274d933b33655e6a642ce3c9569d748d0219 WatchSource:0}: Error finding container 200c6fb9d2f5df0841bf0bcd62ee274d933b33655e6a642ce3c9569d748d0219: Status 404 returned error can't find the container with id 200c6fb9d2f5df0841bf0bcd62ee274d933b33655e6a642ce3c9569d748d0219 Oct 30 22:07:49.146515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.146483 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:49.146588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.146549 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:49.146641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.146584 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:49.505029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.504997 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" event={"ID":"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7","Type":"ContainerStarted","Data":"9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b"} Oct 30 22:07:49.505478 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.505046 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" event={"ID":"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7","Type":"ContainerStarted","Data":"200c6fb9d2f5df0841bf0bcd62ee274d933b33655e6a642ce3c9569d748d0219"} Oct 30 22:07:49.622824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.622769 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podStartSLOduration=2.622749083 podStartE2EDuration="2.622749083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:49.619278852 +0000 UTC m=+1289.918688743" watchObservedRunningTime="2026-10-30 22:07:49.622749083 +0000 UTC m=+1289.922158964" Oct 30 22:07:49.675983 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:49.675937 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:50.675049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:50.675012 4991 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:07:50.675559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:50.675065 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:07:51.673234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:51.673198 4991 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:07:51.673437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:51.673250 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:07:52.673853 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:52.673815 4991 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:07:52.674345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:52.673871 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:07:53.673962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:53.673927 4991 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:07:53.674391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:53.673979 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:07:54.447903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.447850 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs"] Oct 30 22:07:54.460543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.459724 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.503732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.503695 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.503856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.503764 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.503856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.503813 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.503936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.503906 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.503984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.503971 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtg82\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-kube-api-access-rtg82\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.504019 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.504004 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.504094 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.504082 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.504131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.504103 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.504131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.504117 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54859dd1-1236-43c7-bd29-34dc91df57d9-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.560268 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.560218 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs"] Oct 30 22:07:54.605509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605476 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605512 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605719 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605540 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605719 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605681 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtg82\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-kube-api-access-rtg82\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605728 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605833 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605864 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605881 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54859dd1-1236-43c7-bd29-34dc91df57d9-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605912 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.605982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.605926 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.606107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.606070 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.606107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.606095 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.606444 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.606421 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.606571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.606552 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54859dd1-1236-43c7-bd29-34dc91df57d9-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.608064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.608043 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.608333 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.608302 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.674225 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.674187 4991 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:07:54.689725 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.674840 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:07:54.717390 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.710113 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtg82\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-kube-api-access-rtg82\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.732341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.731859 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:54.777751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:54.777716 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:55.260119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.260065 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs"] Oct 30 22:07:55.335540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.335501 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:55.336104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.336080 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:55.336261 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.336246 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:07:55.571199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.571105 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" event={"ID":"54859dd1-1236-43c7-bd29-34dc91df57d9","Type":"ContainerStarted","Data":"8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87"} Oct 30 22:07:55.571199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.571180 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" event={"ID":"54859dd1-1236-43c7-bd29-34dc91df57d9","Type":"ContainerStarted","Data":"2d6abed02dfd9af6ef4d58fd1ad1a9ddcceb3ba59eb70dc0c78cb69a13c416ee"} Oct 30 22:07:55.673927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.673896 4991 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:07:55.674081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.673957 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:07:55.688769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.688724 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podStartSLOduration=1.6887092849999998 podStartE2EDuration="1.688709285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:55.686513708 +0000 UTC m=+1295.985923612" watchObservedRunningTime="2026-10-30 22:07:55.688709285 +0000 UTC m=+1295.988119165" Oct 30 22:07:55.778969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:55.778934 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:07:56.783493 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:56.783447 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:07:56.783889 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:56.783520 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:07:56.818786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:56.818753 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:56.819234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:56.819210 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:56.839869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:56.839795 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:07:57.779334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:57.779299 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:07:57.779515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:57.779361 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:07:58.779641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:58.779607 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:07:58.780027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:58.779663 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:07:59.778856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:59.778757 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:07:59.778856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:07:59.778807 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:08:00.779197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:00.779165 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:08:00.779682 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:00.779658 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:08:01.779258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:01.779222 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:08:01.779698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:01.779286 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:08:03.003795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:03.003755 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:08:03.007379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:03.007356 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:08:03.009629 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:03.009608 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:08:05.082341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:05.078853 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-c77bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:05.082341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:05.078953 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:05.275523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:05.275476 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:05.389398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:05.362700 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:06.278345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:06.277655 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:06.304550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:06.278485 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:08:06.333530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:06.333490 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:08:06.389902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:06.389866 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:06.480340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:06.406770 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:08:06.993471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:06.993435 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:08:07.070343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:07.070296 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:08:12.971830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:12.971794 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85"] Oct 30 22:08:12.976439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:12.976411 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.049340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.045313 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4cf21208-openshift-default-dockercfg-js79h\"" Oct 30 22:08:13.111155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.111106 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tx8h\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-kube-api-access-4tx8h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.111613 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.111575 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.111819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.111799 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.111994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.111957 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.112180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.112160 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.112435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.112417 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.112768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.112743 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.112952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.112936 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.113161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.113147 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.113414 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.113397 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85"] Oct 30 22:08:13.214045 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214011 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214062 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214105 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214151 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214210 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214339 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214431 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tx8h\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-kube-api-access-4tx8h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214473 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214487 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214550 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214583 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.214944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214918 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.215029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.214925 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.215353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.215314 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.218075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.218047 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.222176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.222120 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.279841 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.278516 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tx8h\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-kube-api-access-4tx8h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.280857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.280562 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.293867 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.293612 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:13.749728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.749697 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85"] Oct 30 22:08:13.755959 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:08:13.755927 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ea4e66_c7aa_4acb_81ba_b5cde7ed266f.slice/crio-e0f4f0a40674d1d55cca835af870e5497b8130aa3457a23a9b1fd6e231ec3308 WatchSource:0}: Error finding container e0f4f0a40674d1d55cca835af870e5497b8130aa3457a23a9b1fd6e231ec3308: Status 404 returned error can't find the container with id e0f4f0a40674d1d55cca835af870e5497b8130aa3457a23a9b1fd6e231ec3308 Oct 30 22:08:13.758448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.758410 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:13.758527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.758495 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:13.758563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:13.758526 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:14.350599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:14.350565 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" event={"ID":"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f","Type":"ContainerStarted","Data":"9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb"} Oct 30 22:08:14.351003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:14.350628 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" event={"ID":"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f","Type":"ContainerStarted","Data":"e0f4f0a40674d1d55cca835af870e5497b8130aa3457a23a9b1fd6e231ec3308"} Oct 30 22:08:14.472232 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:14.472170 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" podStartSLOduration=2.472150871 podStartE2EDuration="2.472150871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:14.470038688 +0000 UTC m=+1314.769448569" watchObservedRunningTime="2026-10-30 22:08:14.472150871 +0000 UTC m=+1314.771560753" Oct 30 22:08:15.294872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:15.294838 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:15.301180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:15.301156 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:15.357242 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:15.357213 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:15.357937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:15.357911 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:08:17.913827 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:08:17.913793 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:08:30.094467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:30.094414 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:30.095231 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:30.095192 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:30.737344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:30.721791 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:30.737344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:30.721870 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:30.992273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:30.991692 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:31.078539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:31.078496 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:08:31.283399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:31.277732 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fj82z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:31.283399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:31.277803 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.811718 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.811666 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c"] Oct 30 22:08:37.818078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.818049 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.837086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.837058 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmbm\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-kube-api-access-hxmbm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.837209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.837122 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.837263 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.837204 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.837353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.837309 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.837416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.837364 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.837416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.837382 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.837486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.837464 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/008e673b-52fc-4700-a2ec-769a25faa5f6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.867534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.867504 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:08:37.867641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.867583 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:08:37.867694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.867663 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:08:37.920362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.913735 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7m88w\"" Oct 30 22:08:37.920362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.919744 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c"] Oct 30 22:08:37.938293 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.938261 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmbm\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-kube-api-access-hxmbm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.938475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.938360 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.938475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.938435 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.938622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.938591 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.938695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.938624 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.938695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.938646 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.938695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.938689 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/008e673b-52fc-4700-a2ec-769a25faa5f6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.939723 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.939695 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.941858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.941827 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/008e673b-52fc-4700-a2ec-769a25faa5f6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.942014 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.941995 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.943162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.943137 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.943295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.943274 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.996780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.996751 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:37.996885 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:37.996838 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmbm\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-kube-api-access-hxmbm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qq6c\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:38.130653 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:38.130595 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:38.594981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:38.594946 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c"] Oct 30 22:08:38.601496 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:08:38.601465 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008e673b_52fc_4700_a2ec_769a25faa5f6.slice/crio-cf2d29847a4adc475511100022ce54ff5668a5ea5e4a1a5f947d047985101627 WatchSource:0}: Error finding container cf2d29847a4adc475511100022ce54ff5668a5ea5e4a1a5f947d047985101627: Status 404 returned error can't find the container with id cf2d29847a4adc475511100022ce54ff5668a5ea5e4a1a5f947d047985101627 Oct 30 22:08:38.604264 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:38.604244 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:08:39.101962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:39.101923 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" event={"ID":"008e673b-52fc-4700-a2ec-769a25faa5f6","Type":"ContainerStarted","Data":"cf2d29847a4adc475511100022ce54ff5668a5ea5e4a1a5f947d047985101627"} Oct 30 22:08:47.808750 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:47.808717 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:47.809054 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:47.808780 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:47.921441 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:08:47.921391 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:08:48.169855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:48.169763 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" event={"ID":"008e673b-52fc-4700-a2ec-769a25faa5f6","Type":"ContainerStarted","Data":"2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e"} Oct 30 22:08:48.170024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:48.169925 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:48.171499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:48.171463 4991 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2qq6c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:08:48.171644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:48.171514 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" podUID="008e673b-52fc-4700-a2ec-769a25faa5f6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:08:48.365749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:48.365692 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" podStartSLOduration=2.162748437 podStartE2EDuration="11.365677847s" totalImagesPullingTime="9.20292941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:48.329660592 +0000 UTC m=+1348.629070485" watchObservedRunningTime="2026-10-30 22:08:48.365677847 +0000 UTC m=+1348.665087726" Oct 30 22:08:49.181202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:49.181166 4991 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2qq6c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:08:49.181752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:49.181720 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" podUID="008e673b-52fc-4700-a2ec-769a25faa5f6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:08:52.232652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:52.185100 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:08:53.110628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:53.110584 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:53.396633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:53.396536 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:54.737939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.737902 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp"] Oct 30 22:08:54.743277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.743254 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.750624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750544 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xcp8\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-kube-api-access-2xcp8\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.750701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750649 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf31f51-e90e-460e-8084-1144ef286035-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.750701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750685 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.750816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750708 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf31f51-e90e-460e-8084-1144ef286035-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.750816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750806 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.750924 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750830 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.750924 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750889 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.751030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750943 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.751030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.750973 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.851773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.851740 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.851966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.851789 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.851966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.851818 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.851966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.851884 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xcp8\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-kube-api-access-2xcp8\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.851972 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf31f51-e90e-460e-8084-1144ef286035-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.852009 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.852036 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf31f51-e90e-460e-8084-1144ef286035-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.852135 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.852162 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.852180 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852731 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.852705 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.852925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.852899 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf31f51-e90e-460e-8084-1144ef286035-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.853161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.853133 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.853538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.853517 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.856138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.856093 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.856245 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.856179 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf31f51-e90e-460e-8084-1144ef286035-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.928338 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.924233 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xcp8\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-kube-api-access-2xcp8\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:54.933915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:54.933854 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:55.032399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.030707 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp"] Oct 30 22:08:55.058759 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.058721 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:55.501329 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.501297 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp"] Oct 30 22:08:55.507839 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:08:55.507808 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf31f51_e90e_460e_8084_1144ef286035.slice/crio-593f7540056728fe8d4d63f64058006d3149610f69bf33b7d7e2c0d51c0ce952 WatchSource:0}: Error finding container 593f7540056728fe8d4d63f64058006d3149610f69bf33b7d7e2c0d51c0ce952: Status 404 returned error can't find the container with id 593f7540056728fe8d4d63f64058006d3149610f69bf33b7d7e2c0d51c0ce952 Oct 30 22:08:55.510477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.510435 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:55.510580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.510526 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:55.510580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.510570 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:08:55.646366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.644336 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" event={"ID":"4bf31f51-e90e-460e-8084-1144ef286035","Type":"ContainerStarted","Data":"12c52e9739def98da58a663b4b6c4da0fcc5458c8965b48f5dd9efa4d427e32b"} Oct 30 22:08:55.646366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.644411 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" event={"ID":"4bf31f51-e90e-460e-8084-1144ef286035","Type":"ContainerStarted","Data":"593f7540056728fe8d4d63f64058006d3149610f69bf33b7d7e2c0d51c0ce952"} Oct 30 22:08:55.763454 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:55.763337 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podStartSLOduration=1.763306823 podStartE2EDuration="1.763306823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:55.759221546 +0000 UTC m=+1356.058631426" watchObservedRunningTime="2026-10-30 22:08:55.763306823 +0000 UTC m=+1356.062716705" Oct 30 22:08:56.060077 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:56.059989 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:08:56.061876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:56.061845 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:08:56.062107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:56.062074 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:08:57.059135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:57.059091 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:08:57.059546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:57.059166 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:08:58.059670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:58.059632 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:08:58.060076 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:58.059696 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:08:59.059746 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:59.059714 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:08:59.060128 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:08:59.059773 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:09:00.059810 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:00.059774 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:09:00.060278 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:00.059832 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:09:01.059866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:01.059827 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:09:01.060305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:01.059895 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:09:02.059399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:02.059366 4991 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:09:02.059648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:02.059420 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:09:03.064477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:03.064446 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:09:03.064961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:03.064727 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:09:03.065527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:03.065508 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:09:17.912260 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:09:17.912223 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:09:25.930866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:25.930833 4991 generic.go:356] "Generic (PLEG): container finished" podID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerID="9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb" exitCode=0 Oct 30 22:09:25.931314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:25.930901 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" event={"ID":"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f","Type":"ContainerDied","Data":"9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb"} Oct 30 22:09:25.931387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:25.931371 4991 scope.go:122] "RemoveContainer" containerID="9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb" Oct 30 22:09:25.932856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:25.932825 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:09:25.932977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:25.932909 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:09:25.932977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:25.932957 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:09:26.949347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:26.948591 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" event={"ID":"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f","Type":"ContainerStarted","Data":"8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5"} Oct 30 22:09:27.295043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:27.295010 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:09:27.296306 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:27.296270 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:09:27.296473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:27.296368 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:09:28.295168 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:28.295128 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:09:28.295657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:28.295180 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:09:29.295165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:29.295128 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:09:29.295456 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:29.295429 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:09:30.295455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:30.295372 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:09:30.295455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:30.295427 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:09:30.357229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:30.357192 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:09:31.295448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:31.295407 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:09:31.295944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:31.295478 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:09:32.417800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:32.417765 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:09:32.419135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:09:32.419111 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:09:47.910839 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:09:47.910798 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:10:17.911795 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:10:17.911760 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:10:21.479800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.479769 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295"] Oct 30 22:10:21.484060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.484043 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.539983 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.539949 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxs5k\" (UniqueName: \"kubernetes.io/projected/46023bf7-7cbd-469f-a115-db6165eb2e49-kube-api-access-dxs5k\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.540142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.540122 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.540285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.540262 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46023bf7-7cbd-469f-a115-db6165eb2e49-openshift-service-ca\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.540439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.540368 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-metrics-certs\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.581848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.581817 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dbe0a7-serving-cert\"" Oct 30 22:10:21.585791 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.585763 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295"] Oct 30 22:10:21.641528 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.641491 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-metrics-certs\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.641967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.641617 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxs5k\" (UniqueName: \"kubernetes.io/projected/46023bf7-7cbd-469f-a115-db6165eb2e49-kube-api-access-dxs5k\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.641967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.641725 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.641967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.641841 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46023bf7-7cbd-469f-a115-db6165eb2e49-openshift-service-ca\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.642428 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:10:21.641966 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-serving-cert: secret "maas-api-e2e-api-b-dbe0a7-serving-cert" not found Oct 30 22:10:21.642428 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:10:21.642070 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls podName:46023bf7-7cbd-469f-a115-db6165eb2e49 nodeName:}" failed. No retries permitted until 2026-10-30 22:10:22.142049311 +0000 UTC m=+1442.441459171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls") pod "maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" (UID: "46023bf7-7cbd-469f-a115-db6165eb2e49") : secret "maas-api-e2e-api-b-dbe0a7-serving-cert" not found Oct 30 22:10:21.643225 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.643204 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46023bf7-7cbd-469f-a115-db6165eb2e49-openshift-service-ca\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.647580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.647108 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-metrics-certs\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:21.700124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:21.700094 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxs5k\" (UniqueName: \"kubernetes.io/projected/46023bf7-7cbd-469f-a115-db6165eb2e49-kube-api-access-dxs5k\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:22.148814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:22.148777 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:22.151231 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:22.151199 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls\") pod \"maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:22.399314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:22.399236 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:22.783171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:22.783133 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295"] Oct 30 22:10:22.789506 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:10:22.789472 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46023bf7_7cbd_469f_a115_db6165eb2e49.slice/crio-cf816ba8f640b950e7ff505286a948f72d45bddb14b57843aabfb2b2a50cf2d8 WatchSource:0}: Error finding container cf816ba8f640b950e7ff505286a948f72d45bddb14b57843aabfb2b2a50cf2d8: Status 404 returned error can't find the container with id cf816ba8f640b950e7ff505286a948f72d45bddb14b57843aabfb2b2a50cf2d8 Oct 30 22:10:23.465580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:23.465539 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" event={"ID":"46023bf7-7cbd-469f-a115-db6165eb2e49","Type":"ContainerStarted","Data":"cf816ba8f640b950e7ff505286a948f72d45bddb14b57843aabfb2b2a50cf2d8"} Oct 30 22:10:24.477739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:24.477696 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" event={"ID":"46023bf7-7cbd-469f-a115-db6165eb2e49","Type":"ContainerStarted","Data":"f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b"} Oct 30 22:10:24.478371 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:24.477988 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:24.599343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:24.594265 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" podStartSLOduration=2.642999315 podStartE2EDuration="3.594245716s" totalImagesPullingTime="951.246401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:24.589467179 +0000 UTC m=+1444.888877056" watchObservedRunningTime="2026-10-30 22:10:24.594245716 +0000 UTC m=+1444.893655600" Oct 30 22:10:30.492561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:10:30.492526 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:10:47.926030 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:10:47.925988 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:11:05.134577 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.134504 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc"] Oct 30 22:11:05.138975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.138954 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.232505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.232470 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc"] Oct 30 22:11:05.250366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250305 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250377 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxlnj\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-kube-api-access-vxlnj\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250503 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250671 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250648 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250687 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250762 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250797 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d222ea-b32f-4486-8ef8-4af5acc45280-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250831 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.250955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.250893 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352284 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352244 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352284 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352284 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352357 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352392 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d222ea-b32f-4486-8ef8-4af5acc45280-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352423 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352488 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352573 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352599 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxlnj\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-kube-api-access-vxlnj\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352676 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352682 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.352754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.352737 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.353095 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.353069 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.353431 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.353410 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.353801 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.353772 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d222ea-b32f-4486-8ef8-4af5acc45280-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.355213 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.355188 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.355414 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.355388 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.411036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.410964 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.411310 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.411289 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxlnj\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-kube-api-access-vxlnj\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.452776 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.452747 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:05.887845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.887814 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc"] Oct 30 22:11:05.893651 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:11:05.893613 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d222ea_b32f_4486_8ef8_4af5acc45280.slice/crio-c88da20f871eaf547cad27d0404d320f2edd6bf1a91dd61954d8db11b9d4855d WatchSource:0}: Error finding container c88da20f871eaf547cad27d0404d320f2edd6bf1a91dd61954d8db11b9d4855d: Status 404 returned error can't find the container with id c88da20f871eaf547cad27d0404d320f2edd6bf1a91dd61954d8db11b9d4855d Oct 30 22:11:05.895963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.895932 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:11:05.896056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.895994 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:11:05.896056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:05.896022 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:11:06.866101 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:06.866057 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" event={"ID":"23d222ea-b32f-4486-8ef8-4af5acc45280","Type":"ContainerStarted","Data":"28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79"} Oct 30 22:11:06.866535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:06.866159 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" event={"ID":"23d222ea-b32f-4486-8ef8-4af5acc45280","Type":"ContainerStarted","Data":"c88da20f871eaf547cad27d0404d320f2edd6bf1a91dd61954d8db11b9d4855d"} Oct 30 22:11:07.085337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:07.085271 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podStartSLOduration=2.085258687 podStartE2EDuration="2.085258687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:07.037795235 +0000 UTC m=+1487.337205118" watchObservedRunningTime="2026-10-30 22:11:07.085258687 +0000 UTC m=+1487.384668558" Oct 30 22:11:07.453767 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:07.453726 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:07.455595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:07.455559 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:11:07.455741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:07.455618 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:11:08.453808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:08.453767 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:11:08.454304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:08.453836 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:11:09.453181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:09.453139 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:11:09.453541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:09.453506 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:11:10.453728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:10.453694 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:11:10.454145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:10.453746 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:11:11.453694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:11.453657 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:11:11.454115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:11.454073 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:11:12.453311 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:12.453259 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:11:12.453554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:12.453372 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:11:13.462346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:13.462009 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:11:13.462346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:13.462088 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 22:11:15.464450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:15.464411 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:15.465042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:15.465009 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:16.477347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:16.476600 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:16.477347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:16.476677 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:16.969773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:16.969680 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:16.971163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:16.971142 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:17.031674 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:17.031644 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:11:17.118654 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:17.071484 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:17.353493 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:17.353439 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:17.919792 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:11:17.919524 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:11:20.129289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:20.129261 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:11:20.134298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:20.134271 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:11:38.483260 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.483228 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd"] Oct 30 22:11:38.487539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.487521 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.509661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.509632 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.509860 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.509844 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-metrics-certs\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.509927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.509901 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb60c7ac-8a80-4635-aa1b-d1387a533998-openshift-service-ca\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.510023 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.509999 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kkct\" (UniqueName: \"kubernetes.io/projected/cb60c7ac-8a80-4635-aa1b-d1387a533998-kube-api-access-6kkct\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.586581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.586554 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-161111-serving-cert\"" Oct 30 22:11:38.592866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.592841 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd"] Oct 30 22:11:38.611073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.611042 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.611221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.611201 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-metrics-certs\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.611301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.611279 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb60c7ac-8a80-4635-aa1b-d1387a533998-openshift-service-ca\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.611723 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.611697 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kkct\" (UniqueName: \"kubernetes.io/projected/cb60c7ac-8a80-4635-aa1b-d1387a533998-kube-api-access-6kkct\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.612235 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:11:38.612218 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-161111-serving-cert: secret "maas-api-e2e-api-a-161111-serving-cert" not found Oct 30 22:11:38.612421 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:11:38.612408 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls podName:cb60c7ac-8a80-4635-aa1b-d1387a533998 nodeName:}" failed. No retries permitted until 2026-10-30 22:11:39.11238809 +0000 UTC m=+1519.411797953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls") pod "maas-api-e2e-api-a-161111-68b9495896-tlhgd" (UID: "cb60c7ac-8a80-4635-aa1b-d1387a533998") : secret "maas-api-e2e-api-a-161111-serving-cert" not found Oct 30 22:11:38.612771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.612744 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb60c7ac-8a80-4635-aa1b-d1387a533998-openshift-service-ca\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.615556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.615535 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-metrics-certs\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:38.670113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:38.670087 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kkct\" (UniqueName: \"kubernetes.io/projected/cb60c7ac-8a80-4635-aa1b-d1387a533998-kube-api-access-6kkct\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:39.119387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:39.119354 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:39.121694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:39.121668 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls\") pod \"maas-api-e2e-api-a-161111-68b9495896-tlhgd\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:39.400894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:39.400813 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:39.779218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:39.779176 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd"] Oct 30 22:11:39.785912 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:11:39.785886 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb60c7ac_8a80_4635_aa1b_d1387a533998.slice/crio-21ced342ed3874640dedb9e01d3332cd4c402d85ff24dcb4f1a91ec00e6018b7 WatchSource:0}: Error finding container 21ced342ed3874640dedb9e01d3332cd4c402d85ff24dcb4f1a91ec00e6018b7: Status 404 returned error can't find the container with id 21ced342ed3874640dedb9e01d3332cd4c402d85ff24dcb4f1a91ec00e6018b7 Oct 30 22:11:40.376176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:40.376137 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" event={"ID":"cb60c7ac-8a80-4635-aa1b-d1387a533998","Type":"ContainerStarted","Data":"21ced342ed3874640dedb9e01d3332cd4c402d85ff24dcb4f1a91ec00e6018b7"} Oct 30 22:11:41.405369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:41.403096 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" event={"ID":"cb60c7ac-8a80-4635-aa1b-d1387a533998","Type":"ContainerStarted","Data":"0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5"} Oct 30 22:11:41.405369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:41.403261 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:41.516215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:41.516157 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" podStartSLOduration=2.655498042 podStartE2EDuration="3.516141175s" totalImagesPullingTime="860.643133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:41.515170507 +0000 UTC m=+1521.814580414" watchObservedRunningTime="2026-10-30 22:11:41.516141175 +0000 UTC m=+1521.815551054" Oct 30 22:11:47.414424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:11:47.414394 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:11:47.911869 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:11:47.911830 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:12:17.911452 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:12:17.911398 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:12:47.910830 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:12:47.910793 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:13:05.268309 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:05.268267 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd"] Oct 30 22:13:05.268309 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:05.269622 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" containerID="cri-o://9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b" gracePeriod=30 Oct 30 22:13:10.655416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.655390 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:13:10.807812 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.807722 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-kube-api-access-qqdwh\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-kube-api-access-qqdwh\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.807992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.807846 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-podinfo\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.807992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.807874 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-credential-socket\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.807992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.807892 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-envoy\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.807992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.807965 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istiod-ca-cert\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.808203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808048 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-socket\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.808203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808142 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-data\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.808203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808183 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-credential-socket" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:10.808449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808207 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-token\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-token\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.808449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808302 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-certs\") pod \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\" (UID: \"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7\") " Oct 30 22:13:10.808449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808396 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istiod-ca-cert" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:10.808609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808503 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-socket" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:10.808609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808582 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-data" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:10.808750 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.808733 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-certs" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:10.809033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.809011 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.809109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.809040 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.809109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.809056 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.809109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.809071 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.809109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.809087 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.810575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.810543 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-token" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:10.810575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.810556 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-envoy" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:10.810896 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.810873 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-kube-api-access-qqdwh" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "kube-api-access-qqdwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:10.811485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.811460 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-podinfo" pod "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" (UID: "47b2bf36-34d2-4ac1-8d78-2a9891ee04d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:10.911118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.911082 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.911118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.911119 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdwh\" (UniqueName: \"kubernetes.io/projected/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-kube-api-access-qqdwh\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.911387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.911136 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:10.911387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:10.911151 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:11.260941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.260902 4991 generic.go:356] "Generic (PLEG): container finished" podID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerID="9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b" exitCode=0 Oct 30 22:13:11.261143 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.260955 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" Oct 30 22:13:11.261143 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.260992 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" event={"ID":"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7","Type":"ContainerDied","Data":"9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b"} Oct 30 22:13:11.261143 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.261058 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd" event={"ID":"47b2bf36-34d2-4ac1-8d78-2a9891ee04d7","Type":"ContainerDied","Data":"200c6fb9d2f5df0841bf0bcd62ee274d933b33655e6a642ce3c9569d748d0219"} Oct 30 22:13:11.261143 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.261090 4991 scope.go:122] "RemoveContainer" containerID="9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b" Oct 30 22:13:11.272737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.272716 4991 scope.go:122] "RemoveContainer" containerID="9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b" Oct 30 22:13:11.273063 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:13:11.273045 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b\": container with ID starting with 9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b not found: ID does not exist" containerID="9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b" Oct 30 22:13:11.273136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.273082 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b"} err="failed to get container status \"9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b\": rpc error: code = NotFound desc = could not find container \"9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b\": container with ID starting with 9ac02dd9ba2c230861e62ef94fc1dd2b67a7d67a4c8400b47d527dcba0b8277b not found: ID does not exist" Oct 30 22:13:11.419680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.419647 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd"] Oct 30 22:13:11.464072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.464043 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-qkjcd"] Oct 30 22:13:11.876070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:11.876035 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" path="/var/lib/kubelet/pods/47b2bf36-34d2-4ac1-8d78-2a9891ee04d7/volumes" Oct 30 22:13:17.910981 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:13:17.910942 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:13:31.476985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.476918 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42"] Oct 30 22:13:31.477549 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.477533 4991 cpu_manager.go:414] "containerMap: removing container" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" Oct 30 22:13:31.477596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.477550 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" Oct 30 22:13:31.477673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.477664 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b2bf36-34d2-4ac1-8d78-2a9891ee04d7" containerName="istio-proxy" Oct 30 22:13:31.482912 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.482892 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.532304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.532283 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-427d70-serving-cert\"" Oct 30 22:13:31.582794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.582764 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42"] Oct 30 22:13:31.610373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.610307 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzm5\" (UniqueName: \"kubernetes.io/projected/2e8248e2-029c-4f83-89c0-0d51ace862e0-kube-api-access-pzzm5\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.610527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.610435 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-metrics-certs\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.610589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.610525 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8248e2-029c-4f83-89c0-0d51ace862e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.610760 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.610741 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.712856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.712816 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.713070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.712963 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzm5\" (UniqueName: \"kubernetes.io/projected/2e8248e2-029c-4f83-89c0-0d51ace862e0-kube-api-access-pzzm5\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.713070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.712997 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-metrics-certs\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.713070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.713055 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8248e2-029c-4f83-89c0-0d51ace862e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.713310 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:13:31.713122 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-serving-cert: secret "maas-api-e2e-api-b-427d70-serving-cert" not found Oct 30 22:13:31.713310 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:13:31.713199 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls podName:2e8248e2-029c-4f83-89c0-0d51ace862e0 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:32.213181066 +0000 UTC m=+1632.512590949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls") pod "maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" (UID: "2e8248e2-029c-4f83-89c0-0d51ace862e0") : secret "maas-api-e2e-api-b-427d70-serving-cert" not found Oct 30 22:13:31.713941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.713919 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8248e2-029c-4f83-89c0-0d51ace862e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.733003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.732914 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-metrics-certs\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:31.772113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:31.772081 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzm5\" (UniqueName: \"kubernetes.io/projected/2e8248e2-029c-4f83-89c0-0d51ace862e0-kube-api-access-pzzm5\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:32.221954 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:32.221871 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:32.224232 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:32.224208 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls\") pod \"maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:32.396437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:32.396410 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:32.778765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:32.778731 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42"] Oct 30 22:13:33.445962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:33.445922 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" event={"ID":"2e8248e2-029c-4f83-89c0-0d51ace862e0","Type":"ContainerStarted","Data":"d0ff8170e37f7c1f3494d019e7f43ecbe0b94477a6d53d17a2f5cb910a2fcdef"} Oct 30 22:13:34.463547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:34.463502 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" event={"ID":"2e8248e2-029c-4f83-89c0-0d51ace862e0","Type":"ContainerStarted","Data":"b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde"} Oct 30 22:13:34.463986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:34.463795 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:34.578009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:34.577960 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" podStartSLOduration=2.699242204 podStartE2EDuration="3.577946051s" totalImagesPullingTime="878.703847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:34.577402222 +0000 UTC m=+1634.876812126" watchObservedRunningTime="2026-10-30 22:13:34.577946051 +0000 UTC m=+1634.877355930" Oct 30 22:13:40.477236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:40.477209 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:13:47.910515 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:13:47.910464 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:13:55.366482 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.366440 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv"] Oct 30 22:13:55.367465 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.367406 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" containerID="cri-o://2b61f60d0e9c97ff9b277ee9ea3a45894db15e75c5c9fcc6880f7e3f0c3a6e1c" gracePeriod=30 Oct 30 22:13:55.372036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.372011 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n"] Oct 30 22:13:55.372260 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.372220 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" containerID="cri-o://b9fb7da35156cfeb18ac48f506ad6d48fca082d85bec6df55f40d9b619b746e4" gracePeriod=30 Oct 30 22:13:55.409858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.409832 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp"] Oct 30 22:13:55.410047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.410020 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" containerID="cri-o://a2b111d251ec41c98a8266345db931f60449ff5a12cca21cde1d61743a151c4b" gracePeriod=30 Oct 30 22:13:55.414790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.414764 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5"] Oct 30 22:13:55.415029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:55.415005 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" containerID="cri-o://1e127c34f23913fe20034c10cb02363b8db2557cce83feef610869f6bb3b586c" gracePeriod=30 Oct 30 22:13:59.043281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.043237 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:13:59.043792 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.043311 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:13:59.601444 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.601410 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:13:59.601633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.601477 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:13:59.797622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.797554 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk"] Oct 30 22:13:59.802096 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.802075 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:13:59.900073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.899998 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-04d404-serving-cert\"" Oct 30 22:13:59.903947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.903916 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk"] Oct 30 22:13:59.905705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.905684 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmzk\" (UniqueName: \"kubernetes.io/projected/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-kube-api-access-5lmzk\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:13:59.905804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.905722 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-metrics-certs\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:13:59.905954 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.905931 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:13:59.906053 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:13:59.906008 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.007458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.007427 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-metrics-certs\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.007621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.007513 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.007621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.007584 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.007727 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:00.007638 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-serving-cert: secret "maas-api-e2e-ipp-a-04d404-serving-cert" not found Oct 30 22:14:00.007727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.007688 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmzk\" (UniqueName: \"kubernetes.io/projected/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-kube-api-access-5lmzk\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.007727 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:00.007703 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls podName:a3b68d4d-02e4-43a2-89b5-6d5ef9672446 nodeName:}" failed. No retries permitted until 2026-10-30 22:14:00.507683325 +0000 UTC m=+1660.807093204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls") pod "maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" (UID: "a3b68d4d-02e4-43a2-89b5-6d5ef9672446") : secret "maas-api-e2e-ipp-a-04d404-serving-cert" not found Oct 30 22:14:00.008278 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.008253 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.009885 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.009866 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-metrics-certs\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.070959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.070924 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmzk\" (UniqueName: \"kubernetes.io/projected/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-kube-api-access-5lmzk\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.516254 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.516230 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.518597 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.518569 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.677579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.677502 4991 generic.go:356] "Generic (PLEG): container finished" podID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerID="1e127c34f23913fe20034c10cb02363b8db2557cce83feef610869f6bb3b586c" exitCode=0 Oct 30 22:14:00.677579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.677577 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" event={"ID":"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb","Type":"ContainerDied","Data":"1e127c34f23913fe20034c10cb02363b8db2557cce83feef610869f6bb3b586c"} Oct 30 22:14:00.679458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.679433 4991 generic.go:356] "Generic (PLEG): container finished" podID="13dca42e-3bfb-4105-a04b-f5c813540911" containerID="a2b111d251ec41c98a8266345db931f60449ff5a12cca21cde1d61743a151c4b" exitCode=0 Oct 30 22:14:00.679595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.679514 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" event={"ID":"13dca42e-3bfb-4105-a04b-f5c813540911","Type":"ContainerDied","Data":"a2b111d251ec41c98a8266345db931f60449ff5a12cca21cde1d61743a151c4b"} Oct 30 22:14:00.679595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.679581 4991 scope.go:122] "RemoveContainer" containerID="fd2fdf38e1a0d9e52f76e677a6890c578303f5f25385c4b4f96dc9122ac805bc" Oct 30 22:14:00.682245 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.682225 4991 generic.go:356] "Generic (PLEG): container finished" podID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerID="b9fb7da35156cfeb18ac48f506ad6d48fca082d85bec6df55f40d9b619b746e4" exitCode=0 Oct 30 22:14:00.682366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.682298 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" event={"ID":"c87bb90a-3285-418e-920c-faf02aaf23c6","Type":"ContainerDied","Data":"b9fb7da35156cfeb18ac48f506ad6d48fca082d85bec6df55f40d9b619b746e4"} Oct 30 22:14:00.683959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.683934 4991 generic.go:356] "Generic (PLEG): container finished" podID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerID="2b61f60d0e9c97ff9b277ee9ea3a45894db15e75c5c9fcc6880f7e3f0c3a6e1c" exitCode=0 Oct 30 22:14:00.684050 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.683963 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" event={"ID":"3c8cfc39-4b24-43a7-9871-1388d96d24e0","Type":"ContainerDied","Data":"2b61f60d0e9c97ff9b277ee9ea3a45894db15e75c5c9fcc6880f7e3f0c3a6e1c"} Oct 30 22:14:00.691540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.691519 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:14:00.725215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725173 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-envoy\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.725691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725667 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-podinfo\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.725785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725741 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-certs\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.725785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725774 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-socket\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.725893 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725822 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-credential-socket\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.725893 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725861 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istiod-ca-cert\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.725996 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725888 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-data\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.725996 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.725955 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-kube-api-access-gzw4d\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-kube-api-access-gzw4d\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.726116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.726099 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-token\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-token\") pod \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\" (UID: \"3c8cfc39-4b24-43a7-9871-1388d96d24e0\") " Oct 30 22:14:00.727461 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.727432 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-credential-socket" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.728254 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.728165 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-socket" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.728581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.728252 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istiod-ca-cert" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:00.729237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.728096 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-data" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.729533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.729501 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-envoy" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.729647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.729607 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-certs" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.737605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.736858 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:00.738827 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.738805 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-token" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.739766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.739743 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-podinfo" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:00.740109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.740087 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-kube-api-access-gzw4d" pod "3c8cfc39-4b24-43a7-9871-1388d96d24e0" (UID: "3c8cfc39-4b24-43a7-9871-1388d96d24e0"). InnerVolumeSpecName "kube-api-access-gzw4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.784439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.784417 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 22:14:00.797066 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.797006 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.838673 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-certs\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.838744 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-data\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.838785 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-socket\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.838824 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-podinfo\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.838862 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-socket\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.838952 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-credential-socket\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.838978 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-credential-socket\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839009 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-envoy\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839040 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-envoy\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839070 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-istio-token\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-istio-token\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839103 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-token\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-token\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839177 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13dca42e-3bfb-4105-a04b-f5c813540911-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13dca42e-3bfb-4105-a04b-f5c813540911-istio-podinfo\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839223 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istiod-ca-cert\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839256 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-data\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839358 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13dca42e-3bfb-4105-a04b-f5c813540911-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13dca42e-3bfb-4105-a04b-f5c813540911-istiod-ca-cert\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839446 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-kube-api-access-zmfzv\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-kube-api-access-zmfzv\") pod \"13dca42e-3bfb-4105-a04b-f5c813540911\" (UID: \"13dca42e-3bfb-4105-a04b-f5c813540911\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839477 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-kube-api-access-b6lzb\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-kube-api-access-b6lzb\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.839522 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-certs\") pod \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\" (UID: \"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb\") " Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840118 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840137 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840152 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840167 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840181 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840194 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840207 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840220 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8cfc39-4b24-43a7-9871-1388d96d24e0-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.840936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.840234 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzw4d\" (UniqueName: \"kubernetes.io/projected/3c8cfc39-4b24-43a7-9871-1388d96d24e0-kube-api-access-gzw4d\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.843481 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.842614 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-certs" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.844550 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-data" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.844637 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-credential-socket" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.844741 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istiod-ca-cert" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.845340 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-certs" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.845530 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-socket" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.845692 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-socket" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.845935 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-credential-socket" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.846222 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13dca42e-3bfb-4105-a04b-f5c813540911-istiod-ca-cert" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:00.846941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.846456 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-data" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.854939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.854898 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-kube-api-access-b6lzb" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "kube-api-access-b6lzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.864030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.863175 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-kube-api-access-zmfzv" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "kube-api-access-zmfzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.864030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.863238 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-envoy" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.864030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.863381 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-istio-token" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.864030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.863660 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-envoy" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.864755 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.864714 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-token" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.865362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.865193 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-podinfo" pod "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" (UID: "f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:00.865673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.865625 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13dca42e-3bfb-4105-a04b-f5c813540911-istio-podinfo" pod "13dca42e-3bfb-4105-a04b-f5c813540911" (UID: "13dca42e-3bfb-4105-a04b-f5c813540911"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:00.942193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942120 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfzv\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-kube-api-access-zmfzv\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942148 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6lzb\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-kube-api-access-b6lzb\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942159 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942168 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942177 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942186 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942195 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942204 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942212 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942221 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942229 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942236 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942245 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13dca42e-3bfb-4105-a04b-f5c813540911-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942254 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942262 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13dca42e-3bfb-4105-a04b-f5c813540911-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942270 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942278 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13dca42e-3bfb-4105-a04b-f5c813540911-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.942538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:00.942286 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13dca42e-3bfb-4105-a04b-f5c813540911-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.120466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.120439 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:14:01.207171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.207099 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk"] Oct 30 22:14:01.213428 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:14:01.213399 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b68d4d_02e4_43a2_89b5_6d5ef9672446.slice/crio-5b2d655028e74863e39678e55a99b5143ac8e9b17b2dda0f324bf31abed00765 WatchSource:0}: Error finding container 5b2d655028e74863e39678e55a99b5143ac8e9b17b2dda0f324bf31abed00765: Status 404 returned error can't find the container with id 5b2d655028e74863e39678e55a99b5143ac8e9b17b2dda0f324bf31abed00765 Oct 30 22:14:01.215206 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.215185 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:14:01.263109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263073 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-token\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-token\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263141 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-certs\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263188 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-data\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263273 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-socket\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263357 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-envoy\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263500 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-credential-socket\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263625 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263575 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-kube-api-access-jwhgm\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-kube-api-access-jwhgm\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263625 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263613 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-podinfo\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263722 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263654 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87bb90a-3285-418e-920c-faf02aaf23c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87bb90a-3285-418e-920c-faf02aaf23c6-istiod-ca-cert\") pod \"c87bb90a-3285-418e-920c-faf02aaf23c6\" (UID: \"c87bb90a-3285-418e-920c-faf02aaf23c6\") " Oct 30 22:14:01.263843 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263813 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-socket" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:01.263950 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.263929 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-certs" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:01.264368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.264313 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-data" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:01.264477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.264405 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87bb90a-3285-418e-920c-faf02aaf23c6-istiod-ca-cert" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:01.264477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.264441 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.264477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.264464 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.264646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.264553 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-credential-socket" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:01.267240 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.267205 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-envoy" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:01.267568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.267533 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-kube-api-access-jwhgm" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "kube-api-access-jwhgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:01.268628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.268574 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-podinfo" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:01.270202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.270177 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-token" pod "c87bb90a-3285-418e-920c-faf02aaf23c6" (UID: "c87bb90a-3285-418e-920c-faf02aaf23c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:01.366207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.366171 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.366207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.366202 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.366207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.366213 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwhgm\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-kube-api-access-jwhgm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.366541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.366222 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.366541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.366231 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87bb90a-3285-418e-920c-faf02aaf23c6-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.366541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.366241 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.366541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.366249 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87bb90a-3285-418e-920c-faf02aaf23c6-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.691868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.691839 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" Oct 30 22:14:01.692032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.691833 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n" event={"ID":"c87bb90a-3285-418e-920c-faf02aaf23c6","Type":"ContainerDied","Data":"0f4e63cfc1c5a98eb6011f395d92e5cab30c8205101a4333e883e1ae5ecbf6fc"} Oct 30 22:14:01.692032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.691972 4991 scope.go:122] "RemoveContainer" containerID="b9fb7da35156cfeb18ac48f506ad6d48fca082d85bec6df55f40d9b619b746e4" Oct 30 22:14:01.693293 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.693178 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" event={"ID":"a3b68d4d-02e4-43a2-89b5-6d5ef9672446","Type":"ContainerStarted","Data":"5b2d655028e74863e39678e55a99b5143ac8e9b17b2dda0f324bf31abed00765"} Oct 30 22:14:01.694857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.694827 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" Oct 30 22:14:01.694981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.694834 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv" event={"ID":"3c8cfc39-4b24-43a7-9871-1388d96d24e0","Type":"ContainerDied","Data":"cc75a15e297c96a9ddd0e67001f683e6d00b719a8d3b0db90ade32fddfae019f"} Oct 30 22:14:01.696346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.696330 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" Oct 30 22:14:01.696575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.696551 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5" event={"ID":"f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb","Type":"ContainerDied","Data":"5a557b8e4f40400ed75104a21a51c40cdfad38ca77dcd5abd9d970e284859a95"} Oct 30 22:14:01.697795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.697774 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" event={"ID":"13dca42e-3bfb-4105-a04b-f5c813540911","Type":"ContainerDied","Data":"99e6abada3c96635e31a35a1441fcd2e95fe40e427547b2885e62cca106e703f"} Oct 30 22:14:01.697899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.697841 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp" Oct 30 22:14:01.705828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.705808 4991 scope.go:122] "RemoveContainer" containerID="2b61f60d0e9c97ff9b277ee9ea3a45894db15e75c5c9fcc6880f7e3f0c3a6e1c" Oct 30 22:14:01.723177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.723153 4991 scope.go:122] "RemoveContainer" containerID="1e127c34f23913fe20034c10cb02363b8db2557cce83feef610869f6bb3b586c" Oct 30 22:14:01.740233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.740211 4991 scope.go:122] "RemoveContainer" containerID="a2b111d251ec41c98a8266345db931f60449ff5a12cca21cde1d61743a151c4b" Oct 30 22:14:01.852678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.852644 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n"] Oct 30 22:14:01.899855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.899824 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-k8r6n"] Oct 30 22:14:01.971861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.971765 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp"] Oct 30 22:14:01.977781 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:01.977756 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-tgsmp"] Oct 30 22:14:02.083636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:02.083600 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv"] Oct 30 22:14:02.088289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:02.088261 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-fl2qv"] Oct 30 22:14:02.193792 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:02.193755 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5"] Oct 30 22:14:02.197165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:02.197132 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gq4r5"] Oct 30 22:14:02.707184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:02.707082 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" event={"ID":"a3b68d4d-02e4-43a2-89b5-6d5ef9672446","Type":"ContainerStarted","Data":"e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3"} Oct 30 22:14:02.707184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:02.707177 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:02.821819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:02.821767 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" podStartSLOduration=2.826147683 podStartE2EDuration="3.821752923s" totalImagesPullingTime="995.60524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:02.818253265 +0000 UTC m=+1663.117663157" watchObservedRunningTime="2026-10-30 22:14:02.821752923 +0000 UTC m=+1663.121162802" Oct 30 22:14:03.876609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:03.876579 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" path="/var/lib/kubelet/pods/13dca42e-3bfb-4105-a04b-f5c813540911/volumes" Oct 30 22:14:03.877096 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:03.877081 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" path="/var/lib/kubelet/pods/3c8cfc39-4b24-43a7-9871-1388d96d24e0/volumes" Oct 30 22:14:03.877536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:03.877522 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" path="/var/lib/kubelet/pods/c87bb90a-3285-418e-920c-faf02aaf23c6/volumes" Oct 30 22:14:03.878275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:03.878261 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" path="/var/lib/kubelet/pods/f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb/volumes" Oct 30 22:14:08.722215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:08.722185 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:14:13.679253 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.679216 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl"] Oct 30 22:14:13.679668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.679501 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" containerID="cri-o://4cfc596da50ee1d5a570129180c32df859dca12081ca5df7c7d5045b06e86351" gracePeriod=30 Oct 30 22:14:13.725657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.725629 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd"] Oct 30 22:14:13.725871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.725834 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" containerID="cri-o://37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192" gracePeriod=30 Oct 30 22:14:13.731174 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.731144 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc"] Oct 30 22:14:13.731443 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.731399 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" containerID="cri-o://ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c" gracePeriod=30 Oct 30 22:14:13.770787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.770756 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc"] Oct 30 22:14:13.770953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:13.770933 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" containerID="cri-o://e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380" gracePeriod=30 Oct 30 22:14:16.480663 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:16.480627 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:16.481061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:16.480690 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:17.911637 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:17.911595 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:14:17.921428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:17.921395 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:17.921566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:17.921452 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:17.990687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:17.990649 4991 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:17.990840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:17.990697 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:18.832199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:18.832171 4991 generic.go:356] "Generic (PLEG): container finished" podID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerID="4cfc596da50ee1d5a570129180c32df859dca12081ca5df7c7d5045b06e86351" exitCode=0 Oct 30 22:14:18.832314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:18.832251 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" event={"ID":"5b9630cc-c45d-456a-b76e-84648b0cd3d4","Type":"ContainerDied","Data":"4cfc596da50ee1d5a570129180c32df859dca12081ca5df7c7d5045b06e86351"} Oct 30 22:14:19.056524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.056491 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:14:19.141552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.141515 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:14:19.164418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164387 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-envoy\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164577 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164430 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-podinfo\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164577 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164478 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-data\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164577 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164526 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-certs\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164601 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-token\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-token\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164731 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-kube-api-access-9mz5p\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-kube-api-access-9mz5p\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164763 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-socket\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164905 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-credential-socket\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.164958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.164947 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istiod-ca-cert\") pod \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\" (UID: \"5b9630cc-c45d-456a-b76e-84648b0cd3d4\") " Oct 30 22:14:19.165255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.165217 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-data" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.165656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.165632 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.165748 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.165694 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istiod-ca-cert" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:19.166262 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.166087 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-certs" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.166391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.166298 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-credential-socket" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.166509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.166426 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-socket" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.169163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.169133 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-token" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.170926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.170904 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-kube-api-access-9mz5p" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "kube-api-access-9mz5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.174694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.174672 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-envoy" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.175946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.175921 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-podinfo" pod "5b9630cc-c45d-456a-b76e-84648b0cd3d4" (UID: "5b9630cc-c45d-456a-b76e-84648b0cd3d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:19.180051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.180024 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:14:19.195637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.195618 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266665 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-token\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-token\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266742 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-kube-api-access-bs4hb\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-kube-api-access-bs4hb\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266813 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-socket\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266857 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-podinfo\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266884 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-kube-api-access-gp7bz\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-kube-api-access-gp7bz\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266916 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-envoy\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266955 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-kube-api-access-cngjp\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-kube-api-access-cngjp\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.266986 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-credential-socket\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267039 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-credential-socket\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267078 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-socket\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267127 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-podinfo\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267157 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-credential-socket\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267182 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-envoy\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267216 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-data\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267258 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-envoy\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267292 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-podinfo\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267348 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-token\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267392 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-data\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267420 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-certs\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267471 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-token\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-token\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267515 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-socket\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267551 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d2e12e-be2c-4aea-bf06-4643e567b804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d2e12e-be2c-4aea-bf06-4643e567b804-istiod-ca-cert\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267577 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-data\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267607 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-certs\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267645 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istiod-ca-cert\") pod \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\" (UID: \"1a555122-264a-4159-a3a5-e2e9cb62c5aa\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267687 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istiod-ca-cert\") pod \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\" (UID: \"c1bc7b36-a745-4e9f-975a-4e4cc718ee61\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.267761 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-certs\") pod \"80d2e12e-be2c-4aea-bf06-4643e567b804\" (UID: \"80d2e12e-be2c-4aea-bf06-4643e567b804\") " Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268458 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268477 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mz5p\" (UniqueName: \"kubernetes.io/projected/5b9630cc-c45d-456a-b76e-84648b0cd3d4-kube-api-access-9mz5p\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268489 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268498 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268508 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268517 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268540 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9630cc-c45d-456a-b76e-84648b0cd3d4-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.268805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.268568 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9630cc-c45d-456a-b76e-84648b0cd3d4-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.270580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.270238 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-certs" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.271351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.271284 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-data" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.271864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.271838 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-credential-socket" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.272170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.272150 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-socket" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.272971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.272923 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-data" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.273605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.273313 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-socket" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.273605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.273444 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-token" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.273822 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.273748 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d2e12e-be2c-4aea-bf06-4643e567b804-istiod-ca-cert" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:19.273976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.273927 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-certs" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.274343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.274297 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-credential-socket" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.274444 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.274345 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istiod-ca-cert" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:19.274526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.274515 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-data" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.274824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.274625 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istiod-ca-cert" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:19.274824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.274699 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-credential-socket" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.274961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.274863 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-certs" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.274961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.274934 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-podinfo" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:19.275251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.275197 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-socket" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.276398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.276297 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-token" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.276537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.276426 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-kube-api-access-bs4hb" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "kube-api-access-bs4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.277283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.277255 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-envoy" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.277659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.277635 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-podinfo" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:19.279003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.278141 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-envoy" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.279003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.278201 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-token" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.279003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.278554 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-podinfo" pod "1a555122-264a-4159-a3a5-e2e9cb62c5aa" (UID: "1a555122-264a-4159-a3a5-e2e9cb62c5aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:19.279003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.278983 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-kube-api-access-cngjp" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "kube-api-access-cngjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.279289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.279261 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-kube-api-access-gp7bz" pod "c1bc7b36-a745-4e9f-975a-4e4cc718ee61" (UID: "c1bc7b36-a745-4e9f-975a-4e4cc718ee61"). InnerVolumeSpecName "kube-api-access-gp7bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.279766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.279749 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-envoy" pod "80d2e12e-be2c-4aea-bf06-4643e567b804" (UID: "80d2e12e-be2c-4aea-bf06-4643e567b804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.369623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369588 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369623 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369636 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369650 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4hb\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-kube-api-access-bs4hb\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369664 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369678 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369691 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7bz\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-kube-api-access-gp7bz\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369705 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369717 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cngjp\" (UniqueName: \"kubernetes.io/projected/80d2e12e-be2c-4aea-bf06-4643e567b804-kube-api-access-cngjp\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369730 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369743 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369756 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369769 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369782 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369825 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.369839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369840 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369853 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369868 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369880 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369894 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369908 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc7b36-a745-4e9f-975a-4e4cc718ee61-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369922 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369937 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369953 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d2e12e-be2c-4aea-bf06-4643e567b804-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369966 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d2e12e-be2c-4aea-bf06-4643e567b804-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369979 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a555122-264a-4159-a3a5-e2e9cb62c5aa-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.370340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.369994 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a555122-264a-4159-a3a5-e2e9cb62c5aa-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.842719 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.842684 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" Oct 30 22:14:19.842924 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.842718 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl" event={"ID":"5b9630cc-c45d-456a-b76e-84648b0cd3d4","Type":"ContainerDied","Data":"f7f13a3552096986c57399749e485130472cd8b16dcb264607149caa02f16562"} Oct 30 22:14:19.843154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.843133 4991 scope.go:122] "RemoveContainer" containerID="4cfc596da50ee1d5a570129180c32df859dca12081ca5df7c7d5045b06e86351" Oct 30 22:14:19.844439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.844413 4991 generic.go:356] "Generic (PLEG): container finished" podID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerID="ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c" exitCode=0 Oct 30 22:14:19.844545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.844521 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" event={"ID":"c1bc7b36-a745-4e9f-975a-4e4cc718ee61","Type":"ContainerDied","Data":"ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c"} Oct 30 22:14:19.844589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.844561 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" event={"ID":"c1bc7b36-a745-4e9f-975a-4e4cc718ee61","Type":"ContainerDied","Data":"eee0a478e7808911175f24f7939077bd7538062ea13b1184fb27b989e0a6b8f6"} Oct 30 22:14:19.844691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.844626 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc" Oct 30 22:14:19.846725 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.846684 4991 generic.go:356] "Generic (PLEG): container finished" podID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerID="37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192" exitCode=0 Oct 30 22:14:19.846840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.846798 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" event={"ID":"1a555122-264a-4159-a3a5-e2e9cb62c5aa","Type":"ContainerDied","Data":"37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192"} Oct 30 22:14:19.846840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.846836 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" event={"ID":"1a555122-264a-4159-a3a5-e2e9cb62c5aa","Type":"ContainerDied","Data":"592bb5c3c7b2a1182d0e9d6652061d7180b3ea7ec2f0a0d763b1c96b3dc00b45"} Oct 30 22:14:19.846990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.846912 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd" Oct 30 22:14:19.848984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.848962 4991 generic.go:356] "Generic (PLEG): container finished" podID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerID="e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380" exitCode=0 Oct 30 22:14:19.849096 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.849070 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" event={"ID":"80d2e12e-be2c-4aea-bf06-4643e567b804","Type":"ContainerDied","Data":"e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380"} Oct 30 22:14:19.849201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.849109 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" event={"ID":"80d2e12e-be2c-4aea-bf06-4643e567b804","Type":"ContainerDied","Data":"ddf5027380db737f36b9bfef2a70f1ee7bffa32bd582e8050a395eda693c9b66"} Oct 30 22:14:19.849201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.849181 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc" Oct 30 22:14:19.862256 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.862226 4991 scope.go:122] "RemoveContainer" containerID="ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c" Oct 30 22:14:19.875190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.875168 4991 scope.go:122] "RemoveContainer" containerID="ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c" Oct 30 22:14:19.875565 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:19.875539 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c\": container with ID starting with ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c not found: ID does not exist" containerID="ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c" Oct 30 22:14:19.875650 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.875576 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c"} err="failed to get container status \"ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c\": rpc error: code = NotFound desc = could not find container \"ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c\": container with ID starting with ec4000f1c164d835ea0291553a8bc73a13be57dff06c60804d1e8179e16b0e6c not found: ID does not exist" Oct 30 22:14:19.875650 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.875601 4991 scope.go:122] "RemoveContainer" containerID="37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192" Oct 30 22:14:19.889229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.887375 4991 scope.go:122] "RemoveContainer" containerID="37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192" Oct 30 22:14:19.889229 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:19.887695 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192\": container with ID starting with 37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192 not found: ID does not exist" containerID="37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192" Oct 30 22:14:19.889229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.887728 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192"} err="failed to get container status \"37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192\": rpc error: code = NotFound desc = could not find container \"37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192\": container with ID starting with 37e3b50a39ee2d75557cfb6f1b3ce9a1301f26964baab1a8ba0753a24b814192 not found: ID does not exist" Oct 30 22:14:19.889229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.887750 4991 scope.go:122] "RemoveContainer" containerID="e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380" Oct 30 22:14:19.897188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.897170 4991 scope.go:122] "RemoveContainer" containerID="e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380" Oct 30 22:14:19.897490 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:19.897473 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380\": container with ID starting with e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380 not found: ID does not exist" containerID="e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380" Oct 30 22:14:19.897580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:19.897497 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380"} err="failed to get container status \"e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380\": rpc error: code = NotFound desc = could not find container \"e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380\": container with ID starting with e00da93f65cb960b6b8954f73974eb91302420e94c59a9038abcbf0aab7a2380 not found: ID does not exist" Oct 30 22:14:20.006852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.006823 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc"] Oct 30 22:14:20.055386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.055341 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-27pfc"] Oct 30 22:14:20.115286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.115251 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl"] Oct 30 22:14:20.124125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.124096 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-vfbxl"] Oct 30 22:14:20.228358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.228295 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc"] Oct 30 22:14:20.235075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.235047 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-sw9jc"] Oct 30 22:14:20.340250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.340218 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd"] Oct 30 22:14:20.342668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:20.342643 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-q6zpd"] Oct 30 22:14:21.875864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:21.875824 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" path="/var/lib/kubelet/pods/1a555122-264a-4159-a3a5-e2e9cb62c5aa/volumes" Oct 30 22:14:21.876379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:21.876363 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" path="/var/lib/kubelet/pods/5b9630cc-c45d-456a-b76e-84648b0cd3d4/volumes" Oct 30 22:14:21.876800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:21.876788 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" path="/var/lib/kubelet/pods/80d2e12e-be2c-4aea-bf06-4643e567b804/volumes" Oct 30 22:14:21.877596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:21.877570 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" path="/var/lib/kubelet/pods/c1bc7b36-a745-4e9f-975a-4e4cc718ee61/volumes" Oct 30 22:14:25.544689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:25.544654 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp"] Oct 30 22:14:25.547506 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:25.544890 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" containerID="cri-o://12c52e9739def98da58a663b4b6c4da0fcc5458c8965b48f5dd9efa4d427e32b" gracePeriod=30 Oct 30 22:14:30.923566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:30.923476 4991 generic.go:356] "Generic (PLEG): container finished" podID="4bf31f51-e90e-460e-8084-1144ef286035" containerID="12c52e9739def98da58a663b4b6c4da0fcc5458c8965b48f5dd9efa4d427e32b" exitCode=0 Oct 30 22:14:30.923566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:30.923549 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" event={"ID":"4bf31f51-e90e-460e-8084-1144ef286035","Type":"ContainerDied","Data":"12c52e9739def98da58a663b4b6c4da0fcc5458c8965b48f5dd9efa4d427e32b"} Oct 30 22:14:31.222122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.222097 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:14:31.323804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.323772 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-certs\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.323974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.323822 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-kube-api-access-2xcp8\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-kube-api-access-2xcp8\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.323974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.323880 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bf31f51-e90e-460e-8084-1144ef286035-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf31f51-e90e-460e-8084-1144ef286035-istio-podinfo\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.323974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.323925 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf31f51-e90e-460e-8084-1144ef286035-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf31f51-e90e-460e-8084-1144ef286035-istiod-ca-cert\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.323974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.323962 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-data\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.324197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324053 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-certs" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:31.324197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324107 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-envoy\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.324197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324132 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-istio-token\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-istio-token\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.324377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324222 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-socket\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.324377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324262 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-credential-socket\") pod \"4bf31f51-e90e-460e-8084-1144ef286035\" (UID: \"4bf31f51-e90e-460e-8084-1144ef286035\") " Oct 30 22:14:31.324377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324313 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-data" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:31.324550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324408 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf31f51-e90e-460e-8084-1144ef286035-istiod-ca-cert" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:31.324727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324708 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.324799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324733 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf31f51-e90e-460e-8084-1144ef286035-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.324799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324747 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.325010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.324970 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-socket" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:31.325210 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.325154 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-credential-socket" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:31.326554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.326525 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-kube-api-access-2xcp8" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "kube-api-access-2xcp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:31.326692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.326665 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-istio-token" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:31.326804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.326761 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bf31f51-e90e-460e-8084-1144ef286035-istio-podinfo" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:31.327064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.327042 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-envoy" pod "4bf31f51-e90e-460e-8084-1144ef286035" (UID: "4bf31f51-e90e-460e-8084-1144ef286035"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:31.426244 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.426212 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf31f51-e90e-460e-8084-1144ef286035-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.426244 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.426244 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.426244 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.426257 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.426507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.426271 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.426507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.426284 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf31f51-e90e-460e-8084-1144ef286035-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.426507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.426297 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xcp8\" (UniqueName: \"kubernetes.io/projected/4bf31f51-e90e-460e-8084-1144ef286035-kube-api-access-2xcp8\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:31.930495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.930452 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" event={"ID":"4bf31f51-e90e-460e-8084-1144ef286035","Type":"ContainerDied","Data":"593f7540056728fe8d4d63f64058006d3149610f69bf33b7d7e2c0d51c0ce952"} Oct 30 22:14:31.930495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.930490 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp" Oct 30 22:14:31.931189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:31.930532 4991 scope.go:122] "RemoveContainer" containerID="12c52e9739def98da58a663b4b6c4da0fcc5458c8965b48f5dd9efa4d427e32b" Oct 30 22:14:32.091797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:32.091759 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp"] Oct 30 22:14:32.139870 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:32.139841 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-b8hzp"] Oct 30 22:14:33.875749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:33.875719 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf31f51-e90e-460e-8084-1144ef286035" path="/var/lib/kubelet/pods/4bf31f51-e90e-460e-8084-1144ef286035/volumes" Oct 30 22:14:34.186570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.186495 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295"] Oct 30 22:14:34.186740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.186681 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" podUID="46023bf7-7cbd-469f-a115-db6165eb2e49" containerName="maas-api" containerID="cri-o://f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b" gracePeriod=30 Oct 30 22:14:34.477238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.477213 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:14:34.560474 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.560442 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46023bf7-7cbd-469f-a115-db6165eb2e49-kube-api-access-dxs5k\" (UniqueName: \"kubernetes.io/projected/46023bf7-7cbd-469f-a115-db6165eb2e49-kube-api-access-dxs5k\") pod \"46023bf7-7cbd-469f-a115-db6165eb2e49\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " Oct 30 22:14:34.560634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.560559 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls\") pod \"46023bf7-7cbd-469f-a115-db6165eb2e49\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " Oct 30 22:14:34.560634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.560584 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46023bf7-7cbd-469f-a115-db6165eb2e49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46023bf7-7cbd-469f-a115-db6165eb2e49-openshift-service-ca\") pod \"46023bf7-7cbd-469f-a115-db6165eb2e49\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " Oct 30 22:14:34.560634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.560629 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-metrics-certs\") pod \"46023bf7-7cbd-469f-a115-db6165eb2e49\" (UID: \"46023bf7-7cbd-469f-a115-db6165eb2e49\") " Oct 30 22:14:34.561107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.561076 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46023bf7-7cbd-469f-a115-db6165eb2e49-openshift-service-ca" pod "46023bf7-7cbd-469f-a115-db6165eb2e49" (UID: "46023bf7-7cbd-469f-a115-db6165eb2e49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:34.561460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.561435 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46023bf7-7cbd-469f-a115-db6165eb2e49-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:34.568370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.568301 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46023bf7-7cbd-469f-a115-db6165eb2e49-kube-api-access-dxs5k" pod "46023bf7-7cbd-469f-a115-db6165eb2e49" (UID: "46023bf7-7cbd-469f-a115-db6165eb2e49"). InnerVolumeSpecName "kube-api-access-dxs5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:34.568370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.568330 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls" pod "46023bf7-7cbd-469f-a115-db6165eb2e49" (UID: "46023bf7-7cbd-469f-a115-db6165eb2e49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:34.568550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.568387 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-metrics-certs" pod "46023bf7-7cbd-469f-a115-db6165eb2e49" (UID: "46023bf7-7cbd-469f-a115-db6165eb2e49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:34.663246 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.663205 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxs5k\" (UniqueName: \"kubernetes.io/projected/46023bf7-7cbd-469f-a115-db6165eb2e49-kube-api-access-dxs5k\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:34.663246 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.663247 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:34.663487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.663262 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46023bf7-7cbd-469f-a115-db6165eb2e49-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:34.952128 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.952094 4991 generic.go:356] "Generic (PLEG): container finished" podID="46023bf7-7cbd-469f-a115-db6165eb2e49" containerID="f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b" exitCode=0 Oct 30 22:14:34.952545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.952137 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" Oct 30 22:14:34.952545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.952174 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" event={"ID":"46023bf7-7cbd-469f-a115-db6165eb2e49","Type":"ContainerDied","Data":"f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b"} Oct 30 22:14:34.952545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.952229 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295" event={"ID":"46023bf7-7cbd-469f-a115-db6165eb2e49","Type":"ContainerDied","Data":"cf816ba8f640b950e7ff505286a948f72d45bddb14b57843aabfb2b2a50cf2d8"} Oct 30 22:14:34.952545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.952255 4991 scope.go:122] "RemoveContainer" containerID="f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b" Oct 30 22:14:34.962527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.962507 4991 scope.go:122] "RemoveContainer" containerID="f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b" Oct 30 22:14:34.962812 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:34.962795 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b\": container with ID starting with f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b not found: ID does not exist" containerID="f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b" Oct 30 22:14:34.962887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:34.962821 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b"} err="failed to get container status \"f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b\": rpc error: code = NotFound desc = could not find container \"f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b\": container with ID starting with f8e91ef1d30e36a55a76707b480520cca99f233ca81da6e4d811c4b89cc7363b not found: ID does not exist" Oct 30 22:14:35.126179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:35.125525 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295"] Oct 30 22:14:35.126179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:35.125586 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbe0a7-76ffccc67f-w8295"] Oct 30 22:14:35.875119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:35.875085 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46023bf7-7cbd-469f-a115-db6165eb2e49" path="/var/lib/kubelet/pods/46023bf7-7cbd-469f-a115-db6165eb2e49/volumes" Oct 30 22:14:45.270857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.270816 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g"] Oct 30 22:14:45.271600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271578 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" Oct 30 22:14:45.271600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271600 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271622 4991 cpu_manager.go:414] "containerMap: removing container" podUID="46023bf7-7cbd-469f-a115-db6165eb2e49" containerName="maas-api" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271629 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46023bf7-7cbd-469f-a115-db6165eb2e49" containerName="maas-api" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271641 4991 cpu_manager.go:414] "containerMap: removing container" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271648 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271681 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271688 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271705 4991 cpu_manager.go:414] "containerMap: removing container" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271711 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271737 4991 cpu_manager.go:414] "containerMap: removing container" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271745 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271760 4991 cpu_manager.go:414] "containerMap: removing container" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271767 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" Oct 30 22:14:45.271780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271783 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271790 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271801 4991 cpu_manager.go:414] "containerMap: removing container" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271808 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271820 4991 cpu_manager.go:414] "containerMap: removing container" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271827 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271980 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9630cc-c45d-456a-b76e-84648b0cd3d4" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.271996 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272006 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a555122-264a-4159-a3a5-e2e9cb62c5aa" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272017 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8cfc39-4b24-43a7-9871-1388d96d24e0" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272029 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="46023bf7-7cbd-469f-a115-db6165eb2e49" containerName="maas-api" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272040 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bc7b36-a745-4e9f-975a-4e4cc718ee61" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272050 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d2e12e-be2c-4aea-bf06-4643e567b804" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272060 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf31f51-e90e-460e-8084-1144ef286035" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272068 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87bb90a-3285-418e-920c-faf02aaf23c6" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272078 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" Oct 30 22:14:45.272453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.272090 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12bfbdc-ad52-4cc3-9f8e-8b68603d66eb" containerName="istio-proxy" Oct 30 22:14:45.275962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.275937 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.333341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.326029 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ad7da0-openshift-default-dockercfg-f5fkn\"" Oct 30 22:14:45.434341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.433483 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g"] Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511070 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-envoy\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511116 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384c5fc-2208-42b0-b77c-bab283096d61-istio-podinfo\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511156 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-credential-socket\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511222 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-certs\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511275 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384c5fc-2208-42b0-b77c-bab283096d61-istiod-ca-cert\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511313 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-data\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511425 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-socket\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511485 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-kube-api-access-9hv4t\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.511751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.511537 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-istio-token\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612436 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-certs\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612493 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384c5fc-2208-42b0-b77c-bab283096d61-istiod-ca-cert\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612517 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-data\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612584 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-socket\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612624 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-kube-api-access-9hv4t\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612655 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-istio-token\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612700 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-envoy\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612717 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384c5fc-2208-42b0-b77c-bab283096d61-istio-podinfo\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.612741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.612740 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-credential-socket\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.613365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.613200 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-credential-socket\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.613365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.613263 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-data\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.613365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.613354 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-socket\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.613588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.613456 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-certs\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.613652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.613634 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384c5fc-2208-42b0-b77c-bab283096d61-istiod-ca-cert\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.615386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.615361 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-envoy\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.615801 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.615777 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384c5fc-2208-42b0-b77c-bab283096d61-istio-podinfo\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.677777 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.677744 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-istio-token\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.680762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.680735 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-kube-api-access-9hv4t\") pod \"e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:45.889467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:45.889383 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:46.346563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:46.346527 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g"] Oct 30 22:14:46.351750 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:14:46.351715 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9384c5fc_2208_42b0_b77c_bab283096d61.slice/crio-2f602b500aaff542fc505979776060465fff2e69673446bd780f6c4a692a2dac WatchSource:0}: Error finding container 2f602b500aaff542fc505979776060465fff2e69673446bd780f6c4a692a2dac: Status 404 returned error can't find the container with id 2f602b500aaff542fc505979776060465fff2e69673446bd780f6c4a692a2dac Oct 30 22:14:46.354496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:46.354464 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:46.354592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:46.354531 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:46.354592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:46.354560 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:14:47.066104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:47.066063 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" event={"ID":"9384c5fc-2208-42b0-b77c-bab283096d61","Type":"ContainerStarted","Data":"f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a"} Oct 30 22:14:47.066290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:47.066127 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" event={"ID":"9384c5fc-2208-42b0-b77c-bab283096d61","Type":"ContainerStarted","Data":"2f602b500aaff542fc505979776060465fff2e69673446bd780f6c4a692a2dac"} Oct 30 22:14:47.256229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:47.256170 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" podStartSLOduration=2.256154347 podStartE2EDuration="2.256154347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:47.246427 +0000 UTC m=+1707.545836880" watchObservedRunningTime="2026-10-30 22:14:47.256154347 +0000 UTC m=+1707.555564227" Oct 30 22:14:47.889941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:47.889913 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:47.895633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:47.895610 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:47.911355 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:14:47.911311 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:14:48.074204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:48.074171 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:14:48.074990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:14:48.074962 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:15:00.698483 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.698445 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7"] Oct 30 22:15:00.699085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.699067 4991 cpu_manager.go:414] "containerMap: removing container" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" Oct 30 22:15:00.699085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.699085 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13dca42e-3bfb-4105-a04b-f5c813540911" containerName="istio-proxy" Oct 30 22:15:00.702661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.702641 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.752756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.752732 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4cf21208-serving-cert\"" Oct 30 22:15:00.760163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.760141 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7"] Oct 30 22:15:00.788251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.788222 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmgfq\" (UniqueName: \"kubernetes.io/projected/64173c6c-f886-403e-b581-b1e49114052c-kube-api-access-xmgfq\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.788396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.788265 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.788396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.788371 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-metrics-certs\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.788534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.788430 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64173c6c-f886-403e-b581-b1e49114052c-openshift-service-ca\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.889125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.889097 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64173c6c-f886-403e-b581-b1e49114052c-openshift-service-ca\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.889273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.889207 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmgfq\" (UniqueName: \"kubernetes.io/projected/64173c6c-f886-403e-b581-b1e49114052c-kube-api-access-xmgfq\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.889273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.889242 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.889428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.889278 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-metrics-certs\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.889428 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:15:00.889389 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-serving-cert: secret "maas-api-e2e-mt-4cf21208-serving-cert" not found Oct 30 22:15:00.889505 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:15:00.889462 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls podName:64173c6c-f886-403e-b581-b1e49114052c nodeName:}" failed. No retries permitted until 2026-10-30 22:15:01.389440162 +0000 UTC m=+1721.688850022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls") pod "maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" (UID: "64173c6c-f886-403e-b581-b1e49114052c") : secret "maas-api-e2e-mt-4cf21208-serving-cert" not found Oct 30 22:15:00.890148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.890123 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64173c6c-f886-403e-b581-b1e49114052c-openshift-service-ca\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.891770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.891747 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-metrics-certs\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:00.948356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:00.948294 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmgfq\" (UniqueName: \"kubernetes.io/projected/64173c6c-f886-403e-b581-b1e49114052c-kube-api-access-xmgfq\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:01.397837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:01.397788 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:01.400127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:01.400099 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:01.617554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:01.617525 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:01.996677 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:01.996642 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7"] Oct 30 22:15:02.003133 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:15:02.003079 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64173c6c_f886_403e_b581_b1e49114052c.slice/crio-8f4832db71a780f2d18d46d8f05a04f9d0948976aa417cd7641764209a66e0da WatchSource:0}: Error finding container 8f4832db71a780f2d18d46d8f05a04f9d0948976aa417cd7641764209a66e0da: Status 404 returned error can't find the container with id 8f4832db71a780f2d18d46d8f05a04f9d0948976aa417cd7641764209a66e0da Oct 30 22:15:02.199700 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:02.199663 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" event={"ID":"64173c6c-f886-403e-b581-b1e49114052c","Type":"ContainerStarted","Data":"8f4832db71a780f2d18d46d8f05a04f9d0948976aa417cd7641764209a66e0da"} Oct 30 22:15:03.207365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:03.207284 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" event={"ID":"64173c6c-f886-403e-b581-b1e49114052c","Type":"ContainerStarted","Data":"dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9"} Oct 30 22:15:03.207785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:03.207404 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:03.327236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:03.327185 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" podStartSLOduration=2.380197522 podStartE2EDuration="3.327172355s" totalImagesPullingTime="946.974833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.324230905 +0000 UTC m=+1723.623640802" watchObservedRunningTime="2026-10-30 22:15:03.327172355 +0000 UTC m=+1723.626582234" Oct 30 22:15:09.220304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:09.220273 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:15:17.912373 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:15:17.912307 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:15:47.912809 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:15:47.912774 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:15:50.223305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.223270 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz"] Oct 30 22:15:50.227647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.227626 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.256747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.256715 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db1479-29e2-4366-b1a0-96f85c9957da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.256869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.256790 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdd7l\" (UniqueName: \"kubernetes.io/projected/28db1479-29e2-4366-b1a0-96f85c9957da-kube-api-access-wdd7l\") pod \"payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.283577 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.283548 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz"] Oct 30 22:15:50.325417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.325392 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04d404-dockercfg-gnjfh\"" Oct 30 22:15:50.325541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.325417 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04d404\"" Oct 30 22:15:50.357766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.357736 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db1479-29e2-4366-b1a0-96f85c9957da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.357766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.357768 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdd7l\" (UniqueName: \"kubernetes.io/projected/28db1479-29e2-4366-b1a0-96f85c9957da-kube-api-access-wdd7l\") pod \"payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.358533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.358516 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db1479-29e2-4366-b1a0-96f85c9957da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.415564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.415541 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdd7l\" (UniqueName: \"kubernetes.io/projected/28db1479-29e2-4366-b1a0-96f85c9957da-kube-api-access-wdd7l\") pod \"payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.541607 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.541583 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:50.917608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:50.917524 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz"] Oct 30 22:15:50.924049 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:15:50.924020 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28db1479_29e2_4366_b1a0_96f85c9957da.slice/crio-9518b8b77449e59167b0473e499e8076cf86574a28b485a41e9fa10edd0a9d63 WatchSource:0}: Error finding container 9518b8b77449e59167b0473e499e8076cf86574a28b485a41e9fa10edd0a9d63: Status 404 returned error can't find the container with id 9518b8b77449e59167b0473e499e8076cf86574a28b485a41e9fa10edd0a9d63 Oct 30 22:15:51.576146 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:51.576110 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" event={"ID":"28db1479-29e2-4366-b1a0-96f85c9957da","Type":"ContainerStarted","Data":"9518b8b77449e59167b0473e499e8076cf86574a28b485a41e9fa10edd0a9d63"} Oct 30 22:15:53.593839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:53.593797 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" event={"ID":"28db1479-29e2-4366-b1a0-96f85c9957da","Type":"ContainerStarted","Data":"91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5"} Oct 30 22:15:53.594401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:53.594025 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:15:53.703429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:15:53.703377 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" podStartSLOduration=1.803398214 podStartE2EDuration="3.703363743s" totalImagesPullingTime="1.899965529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:53.702792259 +0000 UTC m=+1774.002202161" watchObservedRunningTime="2026-10-30 22:15:53.703363743 +0000 UTC m=+1774.002773625" Oct 30 22:16:01.749344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.748474 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2"] Oct 30 22:16:01.760037 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.760008 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.787133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.787074 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc"] Oct 30 22:16:01.793018 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.792988 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.848178 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.848147 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2"] Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.902099 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903422 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903479 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3792db35-56ae-4900-afa6-be0b3b222fb3-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903511 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903539 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903569 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903618 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903641 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903700 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903740 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903770 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903945 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.903977 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.904034 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.904060 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5rt\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-kube-api-access-gx5rt\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.904103 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea09944-00ac-4136-80f3-82e2b82f03f5-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.904189 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.904288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.904216 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7d4\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-kube-api-access-zf7d4\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:01.918679 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:01.918646 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc"] Oct 30 22:16:02.005783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.005700 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.005783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.005755 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.005783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.005775 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.005816 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.005832 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.005851 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.005883 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006171 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006227 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006249 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006176 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006283 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5rt\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-kube-api-access-gx5rt\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006303 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006356 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea09944-00ac-4136-80f3-82e2b82f03f5-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006460 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006486 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7d4\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-kube-api-access-zf7d4\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006562 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006585 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006598 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006559 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006642 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3792db35-56ae-4900-afa6-be0b3b222fb3-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006678 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006708 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.006976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006789 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.006976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.006866 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.007288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.007067 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea09944-00ac-4136-80f3-82e2b82f03f5-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.007288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.007069 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.007440 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.007414 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3792db35-56ae-4900-afa6-be0b3b222fb3-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.008992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.008958 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.009204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.009188 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.009308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.009291 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.009549 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.009526 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.067974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.067939 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.068127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.067981 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.068200 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.068164 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5rt\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-kube-api-access-gx5rt\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.089300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.089229 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:02.100218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.100187 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7d4\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-kube-api-access-zf7d4\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.150259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.126572 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:02.562238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.562088 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2"] Oct 30 22:16:02.568133 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:16:02.568098 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea09944_00ac_4136_80f3_82e2b82f03f5.slice/crio-2e60b760a10b5649bf0b534e0f1b9622b48daa38bec3f1227c0dcf6283c19f62 WatchSource:0}: Error finding container 2e60b760a10b5649bf0b534e0f1b9622b48daa38bec3f1227c0dcf6283c19f62: Status 404 returned error can't find the container with id 2e60b760a10b5649bf0b534e0f1b9622b48daa38bec3f1227c0dcf6283c19f62 Oct 30 22:16:02.570594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.570563 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:02.570678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.570627 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:02.570678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.570657 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:02.644736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.640937 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc"] Oct 30 22:16:02.650529 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:16:02.650496 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3792db35_56ae_4900_afa6_be0b3b222fb3.slice/crio-2cc9b864199b555494e424443cf6a74bc99ab977d49ef62add7be5a6cf3adb14 WatchSource:0}: Error finding container 2cc9b864199b555494e424443cf6a74bc99ab977d49ef62add7be5a6cf3adb14: Status 404 returned error can't find the container with id 2cc9b864199b555494e424443cf6a74bc99ab977d49ef62add7be5a6cf3adb14 Oct 30 22:16:02.653405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.653376 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:02.653500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.653440 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:02.653500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.653469 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:02.669826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.669793 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" event={"ID":"3792db35-56ae-4900-afa6-be0b3b222fb3","Type":"ContainerStarted","Data":"2cc9b864199b555494e424443cf6a74bc99ab977d49ef62add7be5a6cf3adb14"} Oct 30 22:16:02.671996 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:02.671968 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" event={"ID":"fea09944-00ac-4136-80f3-82e2b82f03f5","Type":"ContainerStarted","Data":"2e60b760a10b5649bf0b534e0f1b9622b48daa38bec3f1227c0dcf6283c19f62"} Oct 30 22:16:03.706827 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:03.706769 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" event={"ID":"3792db35-56ae-4900-afa6-be0b3b222fb3","Type":"ContainerStarted","Data":"553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c"} Oct 30 22:16:03.708749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:03.708720 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" event={"ID":"fea09944-00ac-4136-80f3-82e2b82f03f5","Type":"ContainerStarted","Data":"d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459"} Oct 30 22:16:04.009984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.009918 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podStartSLOduration=3.009904196 podStartE2EDuration="3.009904196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:04.008516602 +0000 UTC m=+1784.307926483" watchObservedRunningTime="2026-10-30 22:16:04.009904196 +0000 UTC m=+1784.309314081" Oct 30 22:16:04.012568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.012527 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podStartSLOduration=3.012512233 podStartE2EDuration="3.012512233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:03.881920208 +0000 UTC m=+1784.181330090" watchObservedRunningTime="2026-10-30 22:16:04.012512233 +0000 UTC m=+1784.311922114" Oct 30 22:16:04.095566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.095532 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:04.095942 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.095916 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:04.096122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.096095 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:16:04.139075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.139044 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:04.139428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.139405 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:04.139608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.139586 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:16:04.605100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:04.605068 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:16:05.098343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:05.090725 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:05.098343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:05.090802 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:16:05.153344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:05.144485 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:05.153344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:05.144545 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:16:05.186946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:05.186899 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd"] Oct 30 22:16:05.188643 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:05.188613 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" podUID="cb60c7ac-8a80-4635-aa1b-d1387a533998" containerName="maas-api" containerID="cri-o://0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5" gracePeriod=30 Oct 30 22:16:06.100356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.096372 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:06.100356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.096445 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:16:06.148344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.135439 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:06.148344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.135496 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:16:06.612819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.612788 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:16:06.816404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.808574 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls\") pod \"cb60c7ac-8a80-4635-aa1b-d1387a533998\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " Oct 30 22:16:06.816404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.808661 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb60c7ac-8a80-4635-aa1b-d1387a533998-kube-api-access-6kkct\" (UniqueName: \"kubernetes.io/projected/cb60c7ac-8a80-4635-aa1b-d1387a533998-kube-api-access-6kkct\") pod \"cb60c7ac-8a80-4635-aa1b-d1387a533998\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " Oct 30 22:16:06.816404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.808777 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-metrics-certs\") pod \"cb60c7ac-8a80-4635-aa1b-d1387a533998\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " Oct 30 22:16:06.816404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.808840 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb60c7ac-8a80-4635-aa1b-d1387a533998-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb60c7ac-8a80-4635-aa1b-d1387a533998-openshift-service-ca\") pod \"cb60c7ac-8a80-4635-aa1b-d1387a533998\" (UID: \"cb60c7ac-8a80-4635-aa1b-d1387a533998\") " Oct 30 22:16:07.018027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:06.977540 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb60c7ac-8a80-4635-aa1b-d1387a533998-openshift-service-ca" pod "cb60c7ac-8a80-4635-aa1b-d1387a533998" (UID: "cb60c7ac-8a80-4635-aa1b-d1387a533998"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:07.075399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.061864 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb60c7ac-8a80-4635-aa1b-d1387a533998-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:07.119412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.119357 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:07.120332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.120286 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:16:07.128776 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.128750 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:07.128988 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.128964 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:16:07.161506 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.161466 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls" pod "cb60c7ac-8a80-4635-aa1b-d1387a533998" (UID: "cb60c7ac-8a80-4635-aa1b-d1387a533998"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:07.172441 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.171490 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb60c7ac-8a80-4635-aa1b-d1387a533998-kube-api-access-6kkct" pod "cb60c7ac-8a80-4635-aa1b-d1387a533998" (UID: "cb60c7ac-8a80-4635-aa1b-d1387a533998"). InnerVolumeSpecName "kube-api-access-6kkct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:07.172441 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.171580 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-metrics-certs" pod "cb60c7ac-8a80-4635-aa1b-d1387a533998" (UID: "cb60c7ac-8a80-4635-aa1b-d1387a533998"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:07.218034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.217379 4991 generic.go:356] "Generic (PLEG): container finished" podID="cb60c7ac-8a80-4635-aa1b-d1387a533998" containerID="0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5" exitCode=0 Oct 30 22:16:07.218034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.217436 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" event={"ID":"cb60c7ac-8a80-4635-aa1b-d1387a533998","Type":"ContainerDied","Data":"0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5"} Oct 30 22:16:07.218034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.217487 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" event={"ID":"cb60c7ac-8a80-4635-aa1b-d1387a533998","Type":"ContainerDied","Data":"21ced342ed3874640dedb9e01d3332cd4c402d85ff24dcb4f1a91ec00e6018b7"} Oct 30 22:16:07.218034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.217516 4991 scope.go:122] "RemoveContainer" containerID="0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5" Oct 30 22:16:07.218034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.217684 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd" Oct 30 22:16:07.293134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.269902 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:07.295337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.295297 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kkct\" (UniqueName: \"kubernetes.io/projected/cb60c7ac-8a80-4635-aa1b-d1387a533998-kube-api-access-6kkct\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:07.295602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.295587 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb60c7ac-8a80-4635-aa1b-d1387a533998-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:07.367190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.365493 4991 scope.go:122] "RemoveContainer" containerID="0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5" Oct 30 22:16:07.367820 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:07.367741 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5\": container with ID starting with 0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5 not found: ID does not exist" containerID="0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5" Oct 30 22:16:07.367820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.367783 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5"} err="failed to get container status \"0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5\": rpc error: code = NotFound desc = could not find container \"0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5\": container with ID starting with 0d06e13294ef5df151f56f71704ba9bb3ff4c6ff041975c5363ff581d0c5f8a5 not found: ID does not exist" Oct 30 22:16:07.482349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.482301 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc"] Oct 30 22:16:07.500368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.482791 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" containerID="cri-o://28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79" gracePeriod=30 Oct 30 22:16:07.730501 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.639974 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd"] Oct 30 22:16:07.819358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.816452 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-161111-68b9495896-tlhgd"] Oct 30 22:16:07.949191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.949154 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb60c7ac-8a80-4635-aa1b-d1387a533998" path="/var/lib/kubelet/pods/cb60c7ac-8a80-4635-aa1b-d1387a533998/volumes" Oct 30 22:16:07.973639 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.973607 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-967a75"] Oct 30 22:16:07.974743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.974717 4991 cpu_manager.go:414] "containerMap: removing container" podUID="cb60c7ac-8a80-4635-aa1b-d1387a533998" containerName="maas-api" Oct 30 22:16:07.974902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.974888 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb60c7ac-8a80-4635-aa1b-d1387a533998" containerName="maas-api" Oct 30 22:16:07.975217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.975203 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb60c7ac-8a80-4635-aa1b-d1387a533998" containerName="maas-api" Oct 30 22:16:07.994383 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:07.993293 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:08.008337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.003241 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-967a75"] Oct 30 22:16:08.090459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.083113 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr8w\" (UniqueName: \"kubernetes.io/projected/af29ac9b-18a0-499e-9c20-9b4a188a90e5-kube-api-access-gfr8w\") pod \"test-curl-7700-967a75\" (UID: \"af29ac9b-18a0-499e-9c20-9b4a188a90e5\") " pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:08.092678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.092645 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:08.092678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.092665 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmbvp\"" Oct 30 22:16:08.092881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.092700 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:16:08.173834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.173794 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:08.174254 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.173853 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:16:08.185335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.185293 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr8w\" (UniqueName: \"kubernetes.io/projected/af29ac9b-18a0-499e-9c20-9b4a188a90e5-kube-api-access-gfr8w\") pod \"test-curl-7700-967a75\" (UID: \"af29ac9b-18a0-499e-9c20-9b4a188a90e5\") " pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:08.280910 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.280832 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr8w\" (UniqueName: \"kubernetes.io/projected/af29ac9b-18a0-499e-9c20-9b4a188a90e5-kube-api-access-gfr8w\") pod \"test-curl-7700-967a75\" (UID: \"af29ac9b-18a0-499e-9c20-9b4a188a90e5\") " pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:08.318565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.318536 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:08.680407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.680373 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps"] Oct 30 22:16:08.685766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.685744 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.761733 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.760595 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8b4b54-openshift-default-dockercfg-mx5hs\"" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.807784 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-token\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.807837 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-socket\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.807883 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-certs\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.807948 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1911928-972b-4ed5-8bd9-2acd10bb58be-istiod-ca-cert\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.808012 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6kk\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-kube-api-access-6v6kk\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.808091 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-envoy\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.808116 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-podinfo\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.808147 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-credential-socket\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.808223 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-data\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.863367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.826625 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps"] Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.913659 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1911928-972b-4ed5-8bd9-2acd10bb58be-istiod-ca-cert\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.913758 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6kk\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-kube-api-access-6v6kk\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.913861 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-envoy\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.913904 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-podinfo\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.913940 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-credential-socket\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.914053 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-data\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.914204 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-token\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.914238 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-socket\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.914299 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-certs\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.914989 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.914688 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-certs\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.917133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.915442 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-data\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.917133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.915708 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-credential-socket\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.917133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.916640 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1911928-972b-4ed5-8bd9-2acd10bb58be-istiod-ca-cert\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.917133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.916887 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-socket\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:08.990415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:08.989730 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-967a75"] Oct 30 22:16:08.997626 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:16:08.996974 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf29ac9b_18a0_499e_9c20_9b4a188a90e5.slice/crio-f24e2fca6ac201c729462e97c6319d9d901e436cd90e005971b30ed40c41a196 WatchSource:0}: Error finding container f24e2fca6ac201c729462e97c6319d9d901e436cd90e005971b30ed40c41a196: Status 404 returned error can't find the container with id f24e2fca6ac201c729462e97c6319d9d901e436cd90e005971b30ed40c41a196 Oct 30 22:16:09.003641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.003612 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6kk\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-kube-api-access-6v6kk\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:09.003850 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.003784 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-podinfo\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:09.015357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.014888 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-token\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:09.015965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.015904 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-envoy\") pod \"e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:09.057635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.057589 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:09.091929 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.091464 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:09.091929 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.091566 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:16:09.167396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.152233 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:16:09.167396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.152290 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:16:09.296796 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.292820 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-967a75" event={"ID":"af29ac9b-18a0-499e-9c20-9b4a188a90e5","Type":"ContainerStarted","Data":"f24e2fca6ac201c729462e97c6319d9d901e436cd90e005971b30ed40c41a196"} Oct 30 22:16:09.615012 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.614661 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps"] Oct 30 22:16:09.622576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.622252 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:09.622576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.622377 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:09.622576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:09.622433 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:10.094868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.094830 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:10.095455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.095425 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:10.096361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.096308 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:16:10.131907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.131873 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:10.132612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.132465 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:10.133170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.133150 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:16:10.302209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.302170 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" event={"ID":"f1911928-972b-4ed5-8bd9-2acd10bb58be","Type":"ContainerStarted","Data":"04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3"} Oct 30 22:16:10.302672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.302266 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" event={"ID":"f1911928-972b-4ed5-8bd9-2acd10bb58be","Type":"ContainerStarted","Data":"42bba93c1dc7951b8b71ca11f63e5976daab1fd9182e11b2339865b120045a0d"} Oct 30 22:16:10.592028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:10.591963 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podStartSLOduration=2.59194101 podStartE2EDuration="2.59194101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:10.590688224 +0000 UTC m=+1790.890098105" watchObservedRunningTime="2026-10-30 22:16:10.59194101 +0000 UTC m=+1790.891350893" Oct 30 22:16:11.068359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:11.058976 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:11.068359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:11.065219 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:11.313020 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:11.312989 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:11.314529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:11.314136 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:16:12.322818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.322787 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-967a75" event={"ID":"af29ac9b-18a0-499e-9c20-9b4a188a90e5","Type":"ContainerStarted","Data":"6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259"} Oct 30 22:16:12.432449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.432361 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7700-967a75" podStartSLOduration=2.33388084 podStartE2EDuration="5.432347156s" totalImagesPullingTime="3.098466316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:12.429906879 +0000 UTC m=+1792.729316758" watchObservedRunningTime="2026-10-30 22:16:12.432347156 +0000 UTC m=+1792.731757036" Oct 30 22:16:12.693773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.693714 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:16:12.787241 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787210 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-podinfo\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787296 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-token\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-token\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787338 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-credential-socket\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787385 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-envoy\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787444 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d222ea-b32f-4486-8ef8-4af5acc45280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d222ea-b32f-4486-8ef8-4af5acc45280-istiod-ca-cert\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787563 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-kube-api-access-vxlnj\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-kube-api-access-vxlnj\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787627 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-socket\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787639 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-credential-socket" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787691 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-data\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787757 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-certs\") pod \"23d222ea-b32f-4486-8ef8-4af5acc45280\" (UID: \"23d222ea-b32f-4486-8ef8-4af5acc45280\") " Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.787956 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-socket" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788165 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-certs" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788246 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d222ea-b32f-4486-8ef8-4af5acc45280-istiod-ca-cert" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788451 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-data" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788700 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788730 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788744 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788758 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.789624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.788772 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d222ea-b32f-4486-8ef8-4af5acc45280-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.790266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.789710 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-token" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:12.790266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.789729 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-envoy" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:12.790266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.789916 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-podinfo" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:12.790266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.789942 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-kube-api-access-vxlnj" pod "23d222ea-b32f-4486-8ef8-4af5acc45280" (UID: "23d222ea-b32f-4486-8ef8-4af5acc45280"). InnerVolumeSpecName "kube-api-access-vxlnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:12.894956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.894918 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.894956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.894956 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.894956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.894974 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d222ea-b32f-4486-8ef8-4af5acc45280-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.896028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:12.894988 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxlnj\" (UniqueName: \"kubernetes.io/projected/23d222ea-b32f-4486-8ef8-4af5acc45280-kube-api-access-vxlnj\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:13.332972 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.332933 4991 generic.go:356] "Generic (PLEG): container finished" podID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerID="28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79" exitCode=0 Oct 30 22:16:13.333418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.332995 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" Oct 30 22:16:13.333418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.333019 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" event={"ID":"23d222ea-b32f-4486-8ef8-4af5acc45280","Type":"ContainerDied","Data":"28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79"} Oct 30 22:16:13.333418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.333104 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc" event={"ID":"23d222ea-b32f-4486-8ef8-4af5acc45280","Type":"ContainerDied","Data":"c88da20f871eaf547cad27d0404d320f2edd6bf1a91dd61954d8db11b9d4855d"} Oct 30 22:16:13.333418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.333133 4991 scope.go:122] "RemoveContainer" containerID="28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79" Oct 30 22:16:13.335432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.335400 4991 generic.go:356] "Generic (PLEG): container finished" podID="9384c5fc-2208-42b0-b77c-bab283096d61" containerID="f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a" exitCode=0 Oct 30 22:16:13.335551 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.335467 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" event={"ID":"9384c5fc-2208-42b0-b77c-bab283096d61","Type":"ContainerDied","Data":"f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a"} Oct 30 22:16:13.335979 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.335959 4991 scope.go:122] "RemoveContainer" containerID="f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a" Oct 30 22:16:13.337410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.337381 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:13.337529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.337455 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:13.337529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.337491 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:13.345346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.345314 4991 scope.go:122] "RemoveContainer" containerID="28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79" Oct 30 22:16:13.345592 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:13.345570 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79\": container with ID starting with 28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79 not found: ID does not exist" containerID="28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79" Oct 30 22:16:13.345674 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.345598 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79"} err="failed to get container status \"28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79\": rpc error: code = NotFound desc = could not find container \"28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79\": container with ID starting with 28992f8da6cc67c648e87c12618f06b93b49750d2989a724b3e4efce3b189d79 not found: ID does not exist" Oct 30 22:16:13.498310 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.498275 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc"] Oct 30 22:16:13.505636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.505602 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-rbjqc"] Oct 30 22:16:13.876678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:13.876644 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" path="/var/lib/kubelet/pods/23d222ea-b32f-4486-8ef8-4af5acc45280/volumes" Oct 30 22:16:14.056500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:14.056464 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-967a75"] Oct 30 22:16:14.061885 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:14.061853 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-967a75"] Oct 30 22:16:14.355596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:14.355548 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" event={"ID":"9384c5fc-2208-42b0-b77c-bab283096d61","Type":"ContainerStarted","Data":"ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696"} Oct 30 22:16:14.356052 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:14.355746 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7700-967a75" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" containerName="test-curl-7700-967a75" containerID="cri-o://6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259" gracePeriod=2 Oct 30 22:16:14.518269 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:14.518230 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7700-967a75\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:14.566049 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:14.566000 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7700-967a75\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:14.889639 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:14.889607 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:16:14.890624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:14.890593 4991 patch_prober.go:29] interesting pod/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:16:14.890758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:14.890645 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:16:15.890527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:15.890490 4991 patch_prober.go:29] interesting pod/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:16:15.891173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:15.891126 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:16:16.684431 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:16.684405 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:16.782064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:16.782032 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af29ac9b-18a0-499e-9c20-9b4a188a90e5-kube-api-access-gfr8w\" (UniqueName: \"kubernetes.io/projected/af29ac9b-18a0-499e-9c20-9b4a188a90e5-kube-api-access-gfr8w\") pod \"af29ac9b-18a0-499e-9c20-9b4a188a90e5\" (UID: \"af29ac9b-18a0-499e-9c20-9b4a188a90e5\") " Oct 30 22:16:16.784384 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:16.784357 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af29ac9b-18a0-499e-9c20-9b4a188a90e5-kube-api-access-gfr8w" pod "af29ac9b-18a0-499e-9c20-9b4a188a90e5" (UID: "af29ac9b-18a0-499e-9c20-9b4a188a90e5"). InnerVolumeSpecName "kube-api-access-gfr8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:16.884035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:16.883982 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfr8w\" (UniqueName: \"kubernetes.io/projected/af29ac9b-18a0-499e-9c20-9b4a188a90e5-kube-api-access-gfr8w\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:16.890668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:16.890645 4991 patch_prober.go:29] interesting pod/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:16:16.890987 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:16.890699 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:16:17.486118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.486072 4991 generic.go:356] "Generic (PLEG): container finished" podID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" containerID="6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259" exitCode=137 Oct 30 22:16:17.486564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.486420 4991 scope.go:122] "RemoveContainer" containerID="6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259" Oct 30 22:16:17.486865 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.486477 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:17.504441 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.504423 4991 scope.go:122] "RemoveContainer" containerID="6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259" Oct 30 22:16:17.504775 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:17.504757 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259\": container with ID starting with 6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259 not found: ID does not exist" containerID="6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259" Oct 30 22:16:17.504886 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.504869 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259"} err="failed to get container status \"6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259\": rpc error: code = NotFound desc = could not find container \"6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259\": container with ID starting with 6737e8227e5b4f77c99704d2bfa6e44bd608db35f13bdefcbff397579024a259 not found: ID does not exist" Oct 30 22:16:17.553383 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:17.552540 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7700-967a75\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" pod="opendatahub/test-curl-7700-967a75" Oct 30 22:16:17.881674 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.881585 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" path="/var/lib/kubelet/pods/af29ac9b-18a0-499e-9c20-9b4a188a90e5/volumes" Oct 30 22:16:17.890510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.890477 4991 patch_prober.go:29] interesting pod/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:16:17.890679 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:17.890557 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:16:17.993919 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:17.990013 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:16:18.076358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:18.074444 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:16:18.890115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:18.890079 4991 patch_prober.go:29] interesting pod/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 22:16:18.890286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:18.890135 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 22:16:19.895480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:19.895450 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:16:19.896696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:19.896677 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:16:20.187005 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:20.186929 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:16:20.197380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:20.197354 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:16:39.173978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.173946 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t"] Oct 30 22:16:39.174932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.174910 4991 cpu_manager.go:414] "containerMap: removing container" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" containerName="test-curl-7700-967a75" Oct 30 22:16:39.175044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.174934 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" containerName="test-curl-7700-967a75" Oct 30 22:16:39.175044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.174972 4991 cpu_manager.go:414] "containerMap: removing container" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" Oct 30 22:16:39.175044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.174980 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" Oct 30 22:16:39.175203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.175155 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="af29ac9b-18a0-499e-9c20-9b4a188a90e5" containerName="test-curl-7700-967a75" Oct 30 22:16:39.175203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.175176 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d222ea-b32f-4486-8ef8-4af5acc45280" containerName="istio-proxy" Oct 30 22:16:39.182219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.182197 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.237826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.237798 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4576d9-openshift-default-dockercfg-q7cqg\"" Oct 30 22:16:39.270221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270184 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-podinfo\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270311 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-socket\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270416 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-envoy\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270638 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0243d7ec-a527-4410-bb63-5eb4443d234d-istiod-ca-cert\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270708 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270679 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-credential-socket\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270725 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-certs\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270842 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-data\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270877 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj99c\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-kube-api-access-sj99c\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.270927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.270906 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-token\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.281678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.281657 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t"] Oct 30 22:16:39.372562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372525 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-data\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.372562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372557 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj99c\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-kube-api-access-sj99c\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.372784 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372576 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-token\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.372784 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372604 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-podinfo\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.372784 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372640 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-socket\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.372784 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372668 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-envoy\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372822 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0243d7ec-a527-4410-bb63-5eb4443d234d-istiod-ca-cert\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372846 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-credential-socket\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372865 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-certs\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.372983 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-socket\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.373087 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-data\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.373146 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-credential-socket\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.373233 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-certs\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.373531 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.373506 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0243d7ec-a527-4410-bb63-5eb4443d234d-istiod-ca-cert\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.375063 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.375039 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-envoy\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.375286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.375265 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-podinfo\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.431356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.431248 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-token\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.431931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.431904 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj99c\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-kube-api-access-sj99c\") pod \"e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.495535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.495493 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:39.961308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.961271 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t"] Oct 30 22:16:39.967294 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:16:39.967268 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0243d7ec_a527_4410_bb63_5eb4443d234d.slice/crio-a5dc9946b31f36026a71161b198e37da50a31383ae3c590f6db9669f664260bd WatchSource:0}: Error finding container a5dc9946b31f36026a71161b198e37da50a31383ae3c590f6db9669f664260bd: Status 404 returned error can't find the container with id a5dc9946b31f36026a71161b198e37da50a31383ae3c590f6db9669f664260bd Oct 30 22:16:39.969704 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.969673 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:39.969841 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.969740 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:39.969841 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:39.969768 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:16:40.678367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:40.678300 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" event={"ID":"0243d7ec-a527-4410-bb63-5eb4443d234d","Type":"ContainerStarted","Data":"12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d"} Oct 30 22:16:40.678828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:40.678393 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" event={"ID":"0243d7ec-a527-4410-bb63-5eb4443d234d","Type":"ContainerStarted","Data":"a5dc9946b31f36026a71161b198e37da50a31383ae3c590f6db9669f664260bd"} Oct 30 22:16:40.885840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:40.885784 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" podStartSLOduration=1.885770392 podStartE2EDuration="1.885770392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:40.838733706 +0000 UTC m=+1821.138143604" watchObservedRunningTime="2026-10-30 22:16:40.885770392 +0000 UTC m=+1821.185180272" Oct 30 22:16:41.496938 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:41.496910 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:41.502929 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:41.502907 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:41.685837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:41.685808 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:41.686783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:41.686763 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:16:47.782341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:47.781304 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7"] Oct 30 22:16:47.782341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:47.781570 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" podUID="64173c6c-f886-403e-b581-b1e49114052c" containerName="maas-api" containerID="cri-o://dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9" gracePeriod=30 Oct 30 22:16:47.914784 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:47.914747 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:16:48.067828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.067801 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:16:48.111935 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.111902 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls\") pod \"64173c6c-f886-403e-b581-b1e49114052c\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " Oct 30 22:16:48.112073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.111969 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64173c6c-f886-403e-b581-b1e49114052c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64173c6c-f886-403e-b581-b1e49114052c-openshift-service-ca\") pod \"64173c6c-f886-403e-b581-b1e49114052c\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " Oct 30 22:16:48.112073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.111999 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64173c6c-f886-403e-b581-b1e49114052c-kube-api-access-xmgfq\" (UniqueName: \"kubernetes.io/projected/64173c6c-f886-403e-b581-b1e49114052c-kube-api-access-xmgfq\") pod \"64173c6c-f886-403e-b581-b1e49114052c\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " Oct 30 22:16:48.112194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.112179 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-metrics-certs\") pod \"64173c6c-f886-403e-b581-b1e49114052c\" (UID: \"64173c6c-f886-403e-b581-b1e49114052c\") " Oct 30 22:16:48.112413 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.112384 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64173c6c-f886-403e-b581-b1e49114052c-openshift-service-ca" pod "64173c6c-f886-403e-b581-b1e49114052c" (UID: "64173c6c-f886-403e-b581-b1e49114052c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:48.112741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.112720 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64173c6c-f886-403e-b581-b1e49114052c-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:48.114170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.114149 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64173c6c-f886-403e-b581-b1e49114052c-kube-api-access-xmgfq" pod "64173c6c-f886-403e-b581-b1e49114052c" (UID: "64173c6c-f886-403e-b581-b1e49114052c"). InnerVolumeSpecName "kube-api-access-xmgfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:48.114657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.114639 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls" pod "64173c6c-f886-403e-b581-b1e49114052c" (UID: "64173c6c-f886-403e-b581-b1e49114052c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:48.114735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.114658 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-metrics-certs" pod "64173c6c-f886-403e-b581-b1e49114052c" (UID: "64173c6c-f886-403e-b581-b1e49114052c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:48.225343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.218188 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:48.225343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.218222 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64173c6c-f886-403e-b581-b1e49114052c-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:48.225343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.218238 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmgfq\" (UniqueName: \"kubernetes.io/projected/64173c6c-f886-403e-b581-b1e49114052c-kube-api-access-xmgfq\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:48.847342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.844133 4991 generic.go:356] "Generic (PLEG): container finished" podID="64173c6c-f886-403e-b581-b1e49114052c" containerID="dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9" exitCode=0 Oct 30 22:16:48.847342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.844345 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" event={"ID":"64173c6c-f886-403e-b581-b1e49114052c","Type":"ContainerDied","Data":"dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9"} Oct 30 22:16:48.847342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.844403 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" event={"ID":"64173c6c-f886-403e-b581-b1e49114052c","Type":"ContainerDied","Data":"8f4832db71a780f2d18d46d8f05a04f9d0948976aa417cd7641764209a66e0da"} Oct 30 22:16:48.847342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.844437 4991 scope.go:122] "RemoveContainer" containerID="dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9" Oct 30 22:16:48.847342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.844597 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7" Oct 30 22:16:48.879665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.874311 4991 scope.go:122] "RemoveContainer" containerID="dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9" Oct 30 22:16:48.880078 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:16:48.879952 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9\": container with ID starting with dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9 not found: ID does not exist" containerID="dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9" Oct 30 22:16:48.880078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:48.880008 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9"} err="failed to get container status \"dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9\": rpc error: code = NotFound desc = could not find container \"dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9\": container with ID starting with dd5cfe917e69113571fa5635de6ab8638d64c5378f28b88d45cf045b31441eb9 not found: ID does not exist" Oct 30 22:16:49.009965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:49.009924 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7"] Oct 30 22:16:49.052500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:49.052471 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-w96b7"] Oct 30 22:16:49.889635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:16:49.889602 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64173c6c-f886-403e-b581-b1e49114052c" path="/var/lib/kubelet/pods/64173c6c-f886-403e-b581-b1e49114052c/volumes" Oct 30 22:17:17.912796 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:17.912755 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:17:25.000131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.000097 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw"] Oct 30 22:17:25.000833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.000808 4991 cpu_manager.go:414] "containerMap: removing container" podUID="64173c6c-f886-403e-b581-b1e49114052c" containerName="maas-api" Oct 30 22:17:25.000833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.000830 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64173c6c-f886-403e-b581-b1e49114052c" containerName="maas-api" Oct 30 22:17:25.001012 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.000989 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="64173c6c-f886-403e-b581-b1e49114052c" containerName="maas-api" Oct 30 22:17:25.006139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.006119 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.061071 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.061026 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw"] Oct 30 22:17:25.101090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.101058 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ad7da0-dockercfg-ngcns\"" Oct 30 22:17:25.101090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.101088 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ad7da0\"" Oct 30 22:17:25.120833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.120802 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e8c5f9-bdc6-480d-afb5-af949b21b993-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.120960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.120936 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvf4j\" (UniqueName: \"kubernetes.io/projected/c8e8c5f9-bdc6-480d-afb5-af949b21b993-kube-api-access-qvf4j\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.222378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.222344 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvf4j\" (UniqueName: \"kubernetes.io/projected/c8e8c5f9-bdc6-480d-afb5-af949b21b993-kube-api-access-qvf4j\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.222820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.222652 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e8c5f9-bdc6-480d-afb5-af949b21b993-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.223586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.223554 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e8c5f9-bdc6-480d-afb5-af949b21b993-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.280694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.280634 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvf4j\" (UniqueName: \"kubernetes.io/projected/c8e8c5f9-bdc6-480d-afb5-af949b21b993-kube-api-access-qvf4j\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.320254 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.320231 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:25.716622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:25.716552 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw"] Oct 30 22:17:25.723531 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:17:25.723497 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-8205963be698b5be54316379db3876939e8f2c7cb8ac7a0a8832a84a88ff73c9 WatchSource:0}: Error finding container 8205963be698b5be54316379db3876939e8f2c7cb8ac7a0a8832a84a88ff73c9: Status 404 returned error can't find the container with id 8205963be698b5be54316379db3876939e8f2c7cb8ac7a0a8832a84a88ff73c9 Oct 30 22:17:26.227866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:26.227823 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" event={"ID":"c8e8c5f9-bdc6-480d-afb5-af949b21b993","Type":"ContainerStarted","Data":"e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75"} Oct 30 22:17:26.228403 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:26.227881 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" event={"ID":"c8e8c5f9-bdc6-480d-afb5-af949b21b993","Type":"ContainerStarted","Data":"8205963be698b5be54316379db3876939e8f2c7cb8ac7a0a8832a84a88ff73c9"} Oct 30 22:17:26.228787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:26.228764 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:26.341216 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:26.340735 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" podStartSLOduration=2.340714369 podStartE2EDuration="2.340714369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:26.3394743 +0000 UTC m=+1866.638884181" watchObservedRunningTime="2026-10-30 22:17:26.340714369 +0000 UTC m=+1866.640124239" Oct 30 22:17:27.355513 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:27.355447 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:17:27.356013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:27.355521 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:29.255724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:29.255692 4991 generic.go:356] "Generic (PLEG): container finished" podID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerID="04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3" exitCode=0 Oct 30 22:17:29.256147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:29.255752 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" event={"ID":"f1911928-972b-4ed5-8bd9-2acd10bb58be","Type":"ContainerDied","Data":"04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3"} Oct 30 22:17:29.256147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:29.256067 4991 scope.go:122] "RemoveContainer" containerID="04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3" Oct 30 22:17:29.257436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:29.257404 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:29.257506 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:29.257481 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:29.257547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:29.257518 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:30.264117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:30.264083 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" event={"ID":"f1911928-972b-4ed5-8bd9-2acd10bb58be","Type":"ContainerStarted","Data":"11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1"} Oct 30 22:17:31.058489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:31.058455 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:17:31.058662 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:31.058625 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:17:31.058706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:31.058672 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:17:32.058192 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.058160 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:17:32.058627 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.058217 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:17:32.489446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.489412 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng"] Oct 30 22:17:32.493857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.493841 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.555792 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.555740 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4576d9-serving-cert\"" Oct 30 22:17:32.605820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.605785 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng"] Oct 30 22:17:32.619402 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.619370 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.619683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.619660 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87c71bf-75f8-4a27-91cb-1547f7629af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.642344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.619861 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-metrics-certs\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.657845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.657798 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqt7\" (UniqueName: \"kubernetes.io/projected/b87c71bf-75f8-4a27-91cb-1547f7629af6-kube-api-access-jvqt7\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.759223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.759144 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.759399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.759223 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87c71bf-75f8-4a27-91cb-1547f7629af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.759399 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:32.759278 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-serving-cert: secret "maas-api-e2e-ipp-b-4576d9-serving-cert" not found Oct 30 22:17:32.759399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.759312 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-metrics-certs\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.759399 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:32.759356 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls podName:b87c71bf-75f8-4a27-91cb-1547f7629af6 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:33.25933952 +0000 UTC m=+1873.558749379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls") pod "maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" (UID: "b87c71bf-75f8-4a27-91cb-1547f7629af6") : secret "maas-api-e2e-ipp-b-4576d9-serving-cert" not found Oct 30 22:17:32.759399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.759374 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqt7\" (UniqueName: \"kubernetes.io/projected/b87c71bf-75f8-4a27-91cb-1547f7629af6-kube-api-access-jvqt7\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.760037 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.760009 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87c71bf-75f8-4a27-91cb-1547f7629af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.761943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.761920 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-metrics-certs\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:32.817219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:32.817194 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqt7\" (UniqueName: \"kubernetes.io/projected/b87c71bf-75f8-4a27-91cb-1547f7629af6-kube-api-access-jvqt7\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:33.058435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:33.058350 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:17:33.058435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:33.058410 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:17:33.266193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:33.266155 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:33.269288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:33.269259 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:33.413666 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:33.413578 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:33.851168 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:33.851085 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng"] Oct 30 22:17:33.851542 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:17:33.851515 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87c71bf_75f8_4a27_91cb_1547f7629af6.slice/crio-a3c56b2f16cfebd426778dadbc17071983f38d944a8bd6eaa234a90c18d53d0d WatchSource:0}: Error finding container a3c56b2f16cfebd426778dadbc17071983f38d944a8bd6eaa234a90c18d53d0d: Status 404 returned error can't find the container with id a3c56b2f16cfebd426778dadbc17071983f38d944a8bd6eaa234a90c18d53d0d Oct 30 22:17:34.058904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:34.058867 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:17:34.059300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:34.058923 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:17:34.292392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:34.292359 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" event={"ID":"b87c71bf-75f8-4a27-91cb-1547f7629af6","Type":"ContainerStarted","Data":"a3c56b2f16cfebd426778dadbc17071983f38d944a8bd6eaa234a90c18d53d0d"} Oct 30 22:17:35.058605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:35.058527 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:17:35.058605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:35.058574 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 22:17:35.302817 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:35.302779 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" event={"ID":"b87c71bf-75f8-4a27-91cb-1547f7629af6","Type":"ContainerStarted","Data":"fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614"} Oct 30 22:17:35.303354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:35.302925 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:36.062824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:36.062793 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:17:36.063196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:36.063176 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:17:36.063982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:36.063965 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:17:36.235161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:36.235115 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" podStartSLOduration=3.325013441 podStartE2EDuration="4.235101605s" totalImagesPullingTime="910.088164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:35.438641157 +0000 UTC m=+1875.738051037" watchObservedRunningTime="2026-10-30 22:17:36.235101605 +0000 UTC m=+1876.534511488" Oct 30 22:17:38.250726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:38.250700 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:17:39.795634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:39.795603 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc"] Oct 30 22:17:39.796071 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:39.795788 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" containerID="cri-o://553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c" gracePeriod=30 Oct 30 22:17:39.812609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:39.812581 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85"] Oct 30 22:17:39.812772 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:39.812754 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" containerID="cri-o://8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5" gracePeriod=30 Oct 30 22:17:39.829986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:39.829954 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2"] Oct 30 22:17:39.830182 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:39.830160 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" containerID="cri-o://d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459" gracePeriod=30 Oct 30 22:17:40.097880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:40.097800 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:17:40.097880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:40.097869 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:17:40.134698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:40.134666 4991 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:17:40.134920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:40.134892 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:17:41.315876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:41.315849 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:17:45.116611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.116580 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:17:45.187951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.187925 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:17:45.191506 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.191487 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:17:45.208969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.208951 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-certs\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.209053 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.208991 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-kube-api-access-zf7d4\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-kube-api-access-zf7d4\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.209053 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209019 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-socket\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209053 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209042 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-credential-socket\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209086 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-envoy\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209117 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-podinfo\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209159 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-token\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-token\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209184 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-certs\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209217 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-certs" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.209295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209285 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3792db35-56ae-4900-afa6-be0b3b222fb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3792db35-56ae-4900-afa6-be0b3b222fb3-istiod-ca-cert\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.209467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209301 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-socket" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.209467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209380 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istiod-ca-cert\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209398 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-credential-socket" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.209467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209411 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-data\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.209680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209514 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-credential-socket\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.209680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209567 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-socket\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.209680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209597 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-kube-api-access-4tx8h\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-kube-api-access-4tx8h\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.209680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209599 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-data" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.209680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209628 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-podinfo\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.209680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209628 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-certs" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.209680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209668 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-token\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-token\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.210048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209696 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3792db35-56ae-4900-afa6-be0b3b222fb3-istiod-ca-cert" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:45.210048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209709 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-data\") pod \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\" (UID: \"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f\") " Oct 30 22:17:45.210048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209791 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-envoy\") pod \"3792db35-56ae-4900-afa6-be0b3b222fb3\" (UID: \"3792db35-56ae-4900-afa6-be0b3b222fb3\") " Oct 30 22:17:45.210048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209916 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istiod-ca-cert" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:45.210048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.209922 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-socket" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.210275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210123 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-credential-socket" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.210275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210140 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-data" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.210469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210429 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210464 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210482 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210497 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3792db35-56ae-4900-afa6-be0b3b222fb3-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210512 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210525 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210542 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210556 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210569 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.210631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.210583 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.211837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.211807 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-token" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.212209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.212178 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-token" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.212209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.212197 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-kube-api-access-zf7d4" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "kube-api-access-zf7d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.212488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.212419 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-envoy" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.212488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.212474 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-kube-api-access-4tx8h" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "kube-api-access-4tx8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.212672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.212618 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-podinfo" pod "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" (UID: "c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:45.212731 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.212702 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-podinfo" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:45.213770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.213752 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-envoy" pod "3792db35-56ae-4900-afa6-be0b3b222fb3" (UID: "3792db35-56ae-4900-afa6-be0b3b222fb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.311841 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.311804 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-data\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.311941 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-envoy\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.311974 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-socket\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312044 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-token\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-token\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312079 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea09944-00ac-4136-80f3-82e2b82f03f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea09944-00ac-4136-80f3-82e2b82f03f5-istiod-ca-cert\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312131 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-kube-api-access-gx5rt\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-kube-api-access-gx5rt\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312163 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-podinfo\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312256 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-credential-socket\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312362 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-certs\") pod \"fea09944-00ac-4136-80f3-82e2b82f03f5\" (UID: \"fea09944-00ac-4136-80f3-82e2b82f03f5\") " Oct 30 22:17:45.312771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312735 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-socket" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.313028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.312995 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-certs" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.313124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313018 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tx8h\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-kube-api-access-4tx8h\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313065 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313083 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313099 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3792db35-56ae-4900-afa6-be0b3b222fb3-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313130 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7d4\" (UniqueName: \"kubernetes.io/projected/3792db35-56ae-4900-afa6-be0b3b222fb3-kube-api-access-zf7d4\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313145 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313161 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313175 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313206 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.313634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313488 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-credential-socket" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.313940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.313913 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea09944-00ac-4136-80f3-82e2b82f03f5-istiod-ca-cert" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:45.314867 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.314843 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-token" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.315203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.315139 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-data" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.315668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.315643 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-podinfo" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:45.315745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.315668 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-kube-api-access-gx5rt" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "kube-api-access-gx5rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.316422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.316401 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-envoy" pod "fea09944-00ac-4136-80f3-82e2b82f03f5" (UID: "fea09944-00ac-4136-80f3-82e2b82f03f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.374888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.374800 4991 generic.go:356] "Generic (PLEG): container finished" podID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerID="553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c" exitCode=0 Oct 30 22:17:45.374888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.374859 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" Oct 30 22:17:45.375064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.374899 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" event={"ID":"3792db35-56ae-4900-afa6-be0b3b222fb3","Type":"ContainerDied","Data":"553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c"} Oct 30 22:17:45.375064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.374968 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc" event={"ID":"3792db35-56ae-4900-afa6-be0b3b222fb3","Type":"ContainerDied","Data":"2cc9b864199b555494e424443cf6a74bc99ab977d49ef62add7be5a6cf3adb14"} Oct 30 22:17:45.375064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.375005 4991 scope.go:122] "RemoveContainer" containerID="553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c" Oct 30 22:17:45.376507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.376426 4991 generic.go:356] "Generic (PLEG): container finished" podID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerID="d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459" exitCode=0 Oct 30 22:17:45.376852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.376647 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" Oct 30 22:17:45.376852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.376639 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" event={"ID":"fea09944-00ac-4136-80f3-82e2b82f03f5","Type":"ContainerDied","Data":"d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459"} Oct 30 22:17:45.376852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.376788 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2" event={"ID":"fea09944-00ac-4136-80f3-82e2b82f03f5","Type":"ContainerDied","Data":"2e60b760a10b5649bf0b534e0f1b9622b48daa38bec3f1227c0dcf6283c19f62"} Oct 30 22:17:45.378706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.378676 4991 generic.go:356] "Generic (PLEG): container finished" podID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerID="8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5" exitCode=0 Oct 30 22:17:45.378859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.378748 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" event={"ID":"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f","Type":"ContainerDied","Data":"8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5"} Oct 30 22:17:45.378859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.378776 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" event={"ID":"c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f","Type":"ContainerDied","Data":"e0f4f0a40674d1d55cca835af870e5497b8130aa3457a23a9b1fd6e231ec3308"} Oct 30 22:17:45.378859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.378838 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85" Oct 30 22:17:45.388459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.388440 4991 scope.go:122] "RemoveContainer" containerID="553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c" Oct 30 22:17:45.388810 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:45.388792 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c\": container with ID starting with 553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c not found: ID does not exist" containerID="553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c" Oct 30 22:17:45.388882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.388819 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c"} err="failed to get container status \"553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c\": rpc error: code = NotFound desc = could not find container \"553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c\": container with ID starting with 553fa03da132bef1699b8167d3f1ada435c5e1d60f54425a93352f748039436c not found: ID does not exist" Oct 30 22:17:45.388882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.388834 4991 scope.go:122] "RemoveContainer" containerID="d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459" Oct 30 22:17:45.399683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.399605 4991 scope.go:122] "RemoveContainer" containerID="d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459" Oct 30 22:17:45.399913 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:45.399888 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459\": container with ID starting with d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459 not found: ID does not exist" containerID="d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459" Oct 30 22:17:45.400003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.399923 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459"} err="failed to get container status \"d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459\": rpc error: code = NotFound desc = could not find container \"d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459\": container with ID starting with d10d365aa89e00aceda37479958edce4a2b7fcaba58f9f43ddbb5c68bf973459 not found: ID does not exist" Oct 30 22:17:45.400003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.399944 4991 scope.go:122] "RemoveContainer" containerID="8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5" Oct 30 22:17:45.414931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.414905 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.415021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.414936 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.415021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.414951 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.415021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.414965 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.415021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.414979 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.415021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.414993 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea09944-00ac-4136-80f3-82e2b82f03f5-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.415021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.415007 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5rt\" (UniqueName: \"kubernetes.io/projected/fea09944-00ac-4136-80f3-82e2b82f03f5-kube-api-access-gx5rt\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.415219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.415022 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea09944-00ac-4136-80f3-82e2b82f03f5-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.426366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.420793 4991 scope.go:122] "RemoveContainer" containerID="9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb" Oct 30 22:17:45.430215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.430194 4991 scope.go:122] "RemoveContainer" containerID="8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5" Oct 30 22:17:45.430601 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:45.430566 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5\": container with ID starting with 8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5 not found: ID does not exist" containerID="8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5" Oct 30 22:17:45.430697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.430613 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5"} err="failed to get container status \"8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5\": rpc error: code = NotFound desc = could not find container \"8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5\": container with ID starting with 8290b41ef063a7605ad0ffc4688ff6f76d6f4b6f703498d329a6f3f9e59b34b5 not found: ID does not exist" Oct 30 22:17:45.430697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.430636 4991 scope.go:122] "RemoveContainer" containerID="9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb" Oct 30 22:17:45.431007 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:45.430979 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb\": container with ID starting with 9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb not found: ID does not exist" containerID="9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb" Oct 30 22:17:45.431086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.431041 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb"} err="failed to get container status \"9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb\": rpc error: code = NotFound desc = could not find container \"9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb\": container with ID starting with 9a4dad81f5abdc86b184b4700a6cfc7def58b340af567df0d8c560277e5de8cb not found: ID does not exist" Oct 30 22:17:45.617976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.617931 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85"] Oct 30 22:17:45.621819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.621791 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-75d85"] Oct 30 22:17:45.723183 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.723061 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc"] Oct 30 22:17:45.727950 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.727922 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qtbrc"] Oct 30 22:17:45.831329 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.831298 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2"] Oct 30 22:17:45.837044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.837019 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-x4dn2"] Oct 30 22:17:45.875395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.875366 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" path="/var/lib/kubelet/pods/3792db35-56ae-4900-afa6-be0b3b222fb3/volumes" Oct 30 22:17:45.875944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.875921 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" path="/var/lib/kubelet/pods/c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f/volumes" Oct 30 22:17:45.876453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:45.876435 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" path="/var/lib/kubelet/pods/fea09944-00ac-4136-80f3-82e2b82f03f5/volumes" Oct 30 22:17:47.910781 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:47.910746 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:17:50.586986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:50.586941 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42"] Oct 30 22:17:50.587508 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:50.587183 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" podUID="2e8248e2-029c-4f83-89c0-0d51ace862e0" containerName="maas-api" containerID="cri-o://b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde" gracePeriod=30 Oct 30 22:17:51.160493 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.160399 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:17:51.215028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.214997 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-metrics-certs\") pod \"2e8248e2-029c-4f83-89c0-0d51ace862e0\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " Oct 30 22:17:51.215186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.215048 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls\") pod \"2e8248e2-029c-4f83-89c0-0d51ace862e0\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " Oct 30 22:17:51.215186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.215123 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8248e2-029c-4f83-89c0-0d51ace862e0-kube-api-access-pzzm5\" (UniqueName: \"kubernetes.io/projected/2e8248e2-029c-4f83-89c0-0d51ace862e0-kube-api-access-pzzm5\") pod \"2e8248e2-029c-4f83-89c0-0d51ace862e0\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " Oct 30 22:17:51.215314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.215192 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e8248e2-029c-4f83-89c0-0d51ace862e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8248e2-029c-4f83-89c0-0d51ace862e0-openshift-service-ca\") pod \"2e8248e2-029c-4f83-89c0-0d51ace862e0\" (UID: \"2e8248e2-029c-4f83-89c0-0d51ace862e0\") " Oct 30 22:17:51.215619 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.215592 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8248e2-029c-4f83-89c0-0d51ace862e0-openshift-service-ca" pod "2e8248e2-029c-4f83-89c0-0d51ace862e0" (UID: "2e8248e2-029c-4f83-89c0-0d51ace862e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:51.215824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.215803 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8248e2-029c-4f83-89c0-0d51ace862e0-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.217269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.217240 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8248e2-029c-4f83-89c0-0d51ace862e0-kube-api-access-pzzm5" pod "2e8248e2-029c-4f83-89c0-0d51ace862e0" (UID: "2e8248e2-029c-4f83-89c0-0d51ace862e0"). InnerVolumeSpecName "kube-api-access-pzzm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:51.217269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.217251 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-metrics-certs" pod "2e8248e2-029c-4f83-89c0-0d51ace862e0" (UID: "2e8248e2-029c-4f83-89c0-0d51ace862e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:51.217448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.217432 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls" pod "2e8248e2-029c-4f83-89c0-0d51ace862e0" (UID: "2e8248e2-029c-4f83-89c0-0d51ace862e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:51.317470 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.317436 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzzm5\" (UniqueName: \"kubernetes.io/projected/2e8248e2-029c-4f83-89c0-0d51ace862e0-kube-api-access-pzzm5\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.317470 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.317473 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.317718 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.317486 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8248e2-029c-4f83-89c0-0d51ace862e0-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.451405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.447959 4991 generic.go:356] "Generic (PLEG): container finished" podID="2e8248e2-029c-4f83-89c0-0d51ace862e0" containerID="b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde" exitCode=0 Oct 30 22:17:51.451405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.448195 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" event={"ID":"2e8248e2-029c-4f83-89c0-0d51ace862e0","Type":"ContainerDied","Data":"b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde"} Oct 30 22:17:51.451405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.448244 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" event={"ID":"2e8248e2-029c-4f83-89c0-0d51ace862e0","Type":"ContainerDied","Data":"d0ff8170e37f7c1f3494d019e7f43ecbe0b94477a6d53d17a2f5cb910a2fcdef"} Oct 30 22:17:51.451405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.448273 4991 scope.go:122] "RemoveContainer" containerID="b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde" Oct 30 22:17:51.451405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.448424 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42" Oct 30 22:17:51.494512 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.494469 4991 scope.go:122] "RemoveContainer" containerID="b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde" Oct 30 22:17:51.497717 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:17:51.497673 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde\": container with ID starting with b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde not found: ID does not exist" containerID="b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde" Oct 30 22:17:51.497965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.497936 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde"} err="failed to get container status \"b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde\": rpc error: code = NotFound desc = could not find container \"b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde\": container with ID starting with b82788542048cd3fa01d5fad22809e73ebbb71ff999282baf103380efa9d2bde not found: ID does not exist" Oct 30 22:17:51.652446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.646390 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42"] Oct 30 22:17:51.652446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.648230 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-427d70-6cd6c87b9d-s9f42"] Oct 30 22:17:51.882472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:51.879153 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e8248e2-029c-4f83-89c0-0d51ace862e0" path="/var/lib/kubelet/pods/2e8248e2-029c-4f83-89c0-0d51ace862e0/volumes" Oct 30 22:17:56.490619 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:56.490575 4991 generic.go:356] "Generic (PLEG): container finished" podID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerID="12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d" exitCode=0 Oct 30 22:17:56.491097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:56.490621 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" event={"ID":"0243d7ec-a527-4410-bb63-5eb4443d234d","Type":"ContainerDied","Data":"12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d"} Oct 30 22:17:56.491169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:56.491151 4991 scope.go:122] "RemoveContainer" containerID="12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d" Oct 30 22:17:56.492622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:56.492593 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:56.492696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:56.492660 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:56.492696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:56.492690 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:17:56.686100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:56.686066 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:17:57.136342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:57.132050 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs"] Oct 30 22:17:57.136342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:57.132276 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" containerID="cri-o://8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87" gracePeriod=30 Oct 30 22:17:57.504435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:57.504404 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" event={"ID":"0243d7ec-a527-4410-bb63-5eb4443d234d","Type":"ContainerStarted","Data":"95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7"} Oct 30 22:17:58.496936 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:58.496902 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:17:58.497908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:58.497880 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:17:58.498039 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:58.497928 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:17:59.496810 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:59.496774 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:17:59.497215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:17:59.496829 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:18:00.497019 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:00.496984 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:18:00.497493 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:00.497421 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:18:01.496853 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:01.496815 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:18:01.497021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:01.496872 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:18:02.617900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.587210 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:18:02.617900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.587303 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:18:02.617900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.601414 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:18:02.625895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.625859 4991 generic.go:356] "Generic (PLEG): container finished" podID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerID="8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87" exitCode=0 Oct 30 22:18:02.626192 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.626165 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" Oct 30 22:18:02.626769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.626014 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" event={"ID":"54859dd1-1236-43c7-bd29-34dc91df57d9","Type":"ContainerDied","Data":"8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87"} Oct 30 22:18:02.627007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.626988 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs" event={"ID":"54859dd1-1236-43c7-bd29-34dc91df57d9","Type":"ContainerDied","Data":"2d6abed02dfd9af6ef4d58fd1ad1a9ddcceb3ba59eb70dc0c78cb69a13c416ee"} Oct 30 22:18:02.627574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.627285 4991 scope.go:122] "RemoveContainer" containerID="8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87" Oct 30 22:18:02.642842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.642810 4991 scope.go:122] "RemoveContainer" containerID="8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87" Oct 30 22:18:02.643224 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:02.643203 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87\": container with ID starting with 8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87 not found: ID does not exist" containerID="8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87" Oct 30 22:18:02.643307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.643234 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87"} err="failed to get container status \"8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87\": rpc error: code = NotFound desc = could not find container \"8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87\": container with ID starting with 8cbff5eef063726f8abec0e9f7fd69dbfc0ac0fe98dd58be0edd2f0bc1e70b87 not found: ID does not exist" Oct 30 22:18:02.713553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713459 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-credential-socket\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.713553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713522 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-data\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.713783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713652 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-token\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-token\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.713783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713695 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-podinfo\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.713783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713753 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54859dd1-1236-43c7-bd29-34dc91df57d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54859dd1-1236-43c7-bd29-34dc91df57d9-istiod-ca-cert\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.713940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713783 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-data" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:02.713940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713796 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-credential-socket" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:02.713940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713806 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-kube-api-access-rtg82\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-kube-api-access-rtg82\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.713940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713899 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-socket\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.714157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.713959 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-envoy\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.714157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714013 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-certs\") pod \"54859dd1-1236-43c7-bd29-34dc91df57d9\" (UID: \"54859dd1-1236-43c7-bd29-34dc91df57d9\") " Oct 30 22:18:02.714157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714142 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54859dd1-1236-43c7-bd29-34dc91df57d9-istiod-ca-cert" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:02.714489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714367 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-certs" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:02.714831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714648 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54859dd1-1236-43c7-bd29-34dc91df57d9-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.714831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714674 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.714831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714688 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.714831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714703 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.714831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.714777 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-socket" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:02.717000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.716731 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-podinfo" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:02.718458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.718427 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-envoy" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:02.718713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.718690 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-token" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:02.718713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.718700 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-kube-api-access-rtg82" pod "54859dd1-1236-43c7-bd29-34dc91df57d9" (UID: "54859dd1-1236-43c7-bd29-34dc91df57d9"). InnerVolumeSpecName "kube-api-access-rtg82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:02.815580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.815542 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.815580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.815574 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.815795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.815588 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtg82\" (UniqueName: \"kubernetes.io/projected/54859dd1-1236-43c7-bd29-34dc91df57d9-kube-api-access-rtg82\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.815795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.815603 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:02.815795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:02.815615 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54859dd1-1236-43c7-bd29-34dc91df57d9-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:03.048207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:03.048141 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs"] Oct 30 22:18:03.051525 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:03.051497 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-7xbfs"] Oct 30 22:18:03.500469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:03.500440 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:18:03.500969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:03.500948 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:18:03.501705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:03.501687 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:18:03.880686 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:03.880610 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" path="/var/lib/kubelet/pods/54859dd1-1236-43c7-bd29-34dc91df57d9/volumes" Oct 30 22:18:11.853194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853162 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f"] Oct 30 22:18:11.853871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853851 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853872 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853883 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853890 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853903 4991 cpu_manager.go:414] "containerMap: removing container" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853908 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853918 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2e8248e2-029c-4f83-89c0-0d51ace862e0" containerName="maas-api" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853925 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e8248e2-029c-4f83-89c0-0d51ace862e0" containerName="maas-api" Oct 30 22:18:11.853947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853941 4991 cpu_manager.go:414] "containerMap: removing container" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853950 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853961 4991 cpu_manager.go:414] "containerMap: removing container" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.853968 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.854098 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.854116 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="54859dd1-1236-43c7-bd29-34dc91df57d9" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.854124 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e8248e2-029c-4f83-89c0-0d51ace862e0" containerName="maas-api" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.854132 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ea4e66-c7aa-4acb-81ba-b5cde7ed266f" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.854140 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea09944-00ac-4136-80f3-82e2b82f03f5" containerName="istio-proxy" Oct 30 22:18:11.854258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.854146 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3792db35-56ae-4900-afa6-be0b3b222fb3" containerName="istio-proxy" Oct 30 22:18:11.857508 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.857490 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:11.909765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.909731 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8daffd-serving-cert\"" Oct 30 22:18:11.913790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.913769 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f"] Oct 30 22:18:11.921859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.921833 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45fdf4a4-9485-4182-adb6-0352a25f3a1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:11.922016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.921875 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:11.922016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.921935 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:11.922016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:11.921987 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56m9\" (UniqueName: \"kubernetes.io/projected/45fdf4a4-9485-4182-adb6-0352a25f3a1e-kube-api-access-f56m9\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.023415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.023352 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.023588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.023462 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56m9\" (UniqueName: \"kubernetes.io/projected/45fdf4a4-9485-4182-adb6-0352a25f3a1e-kube-api-access-f56m9\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.023588 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:12.023563 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-serving-cert: secret "maas-api-e2e-ipp-a-8daffd-serving-cert" not found Oct 30 22:18:12.023705 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:12.023633 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls podName:45fdf4a4-9485-4182-adb6-0352a25f3a1e nodeName:}" failed. No retries permitted until 2026-10-30 22:18:12.523611843 +0000 UTC m=+1912.823021702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls") pod "maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" (UID: "45fdf4a4-9485-4182-adb6-0352a25f3a1e") : secret "maas-api-e2e-ipp-a-8daffd-serving-cert" not found Oct 30 22:18:12.023705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.023692 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45fdf4a4-9485-4182-adb6-0352a25f3a1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.023811 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.023721 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.024646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.024621 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45fdf4a4-9485-4182-adb6-0352a25f3a1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.026259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.026233 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.091951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.091852 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56m9\" (UniqueName: \"kubernetes.io/projected/45fdf4a4-9485-4182-adb6-0352a25f3a1e-kube-api-access-f56m9\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.531206 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.531174 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.533819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.533793 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:12.773819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:12.773771 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:13.159051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:13.158972 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f"] Oct 30 22:18:13.165291 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:18:13.165261 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fdf4a4_9485_4182_adb6_0352a25f3a1e.slice/crio-6b1dccacf6a4f250ae44a9302301fc52744f22d7f472b566ad01b7d8b901329b WatchSource:0}: Error finding container 6b1dccacf6a4f250ae44a9302301fc52744f22d7f472b566ad01b7d8b901329b: Status 404 returned error can't find the container with id 6b1dccacf6a4f250ae44a9302301fc52744f22d7f472b566ad01b7d8b901329b Oct 30 22:18:13.709569 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:13.709530 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" event={"ID":"45fdf4a4-9485-4182-adb6-0352a25f3a1e","Type":"ContainerStarted","Data":"6b1dccacf6a4f250ae44a9302301fc52744f22d7f472b566ad01b7d8b901329b"} Oct 30 22:18:14.736626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:14.736581 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" event={"ID":"45fdf4a4-9485-4182-adb6-0352a25f3a1e","Type":"ContainerStarted","Data":"432588e8034b76c7a75c397c473e119288e82eed6583939c1fe4edc77c3d5ccc"} Oct 30 22:18:14.738664 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:14.738639 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:14.852584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:14.852513 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" podStartSLOduration=2.82050945 podStartE2EDuration="3.852493203s" totalImagesPullingTime="1.031983753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:14.850151511 +0000 UTC m=+1915.149561411" watchObservedRunningTime="2026-10-30 22:18:14.852493203 +0000 UTC m=+1915.151903083" Oct 30 22:18:17.918196 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:17.918157 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:18:19.859343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:19.858647 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq"] Oct 30 22:18:19.859343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:19.858883 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" containerID="cri-o://475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b" gracePeriod=30 Oct 30 22:18:19.998344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:19.964381 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn"] Oct 30 22:18:19.998344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:19.964642 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" containerID="cri-o://8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7" gracePeriod=30 Oct 30 22:18:20.293487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:20.293453 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6"] Oct 30 22:18:20.297353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:20.294028 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" containerID="cri-o://f21965bc286bf80fe5acbe19e4e94df7a72fbe48344788ab35466ae231554fca" gracePeriod=30 Oct 30 22:18:20.803344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:20.779452 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z"] Oct 30 22:18:20.864610 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:20.864541 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" containerID="cri-o://4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e" gracePeriod=30 Oct 30 22:18:21.756855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:21.756820 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:18:25.391272 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:25.391226 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x2fbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:18:25.391762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:25.391304 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:18:25.475618 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:25.475586 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-fpmc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:18:25.475761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:25.475643 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:18:25.976609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:25.976568 4991 generic.go:356] "Generic (PLEG): container finished" podID="b2b68edd-8617-43ae-9430-6616068a1698" containerID="f21965bc286bf80fe5acbe19e4e94df7a72fbe48344788ab35466ae231554fca" exitCode=0 Oct 30 22:18:25.976795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:25.976691 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" event={"ID":"b2b68edd-8617-43ae-9430-6616068a1698","Type":"ContainerDied","Data":"f21965bc286bf80fe5acbe19e4e94df7a72fbe48344788ab35466ae231554fca"} Oct 30 22:18:26.015855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.015830 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:18:26.107631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.107596 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-istio-token\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-istio-token\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.107821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.107656 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b68edd-8617-43ae-9430-6616068a1698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b68edd-8617-43ae-9430-6616068a1698-istiod-ca-cert\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.107821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.107691 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-socket\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.107821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.107725 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-envoy\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.107821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.107796 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2b68edd-8617-43ae-9430-6616068a1698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b68edd-8617-43ae-9430-6616068a1698-istio-podinfo\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.108062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.107874 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-data\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.108062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.107912 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-credential-socket\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.108393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.108286 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-data" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.108393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.108308 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-credential-socket" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.108393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.108364 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-socket" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.108659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.108205 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b68edd-8617-43ae-9430-6616068a1698-istiod-ca-cert" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:26.108830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.108805 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-certs\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.109173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.109152 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-certs" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.109261 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.109215 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-kube-api-access-bxnnt\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-kube-api-access-bxnnt\") pod \"b2b68edd-8617-43ae-9430-6616068a1698\" (UID: \"b2b68edd-8617-43ae-9430-6616068a1698\") " Oct 30 22:18:26.110056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110035 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.110214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110200 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.110351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110302 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.110351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110340 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.110518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110356 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b68edd-8617-43ae-9430-6616068a1698-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.110591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110564 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-istio-token" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.110785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110760 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-envoy" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.110944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.110920 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2b68edd-8617-43ae-9430-6616068a1698-istio-podinfo" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:26.111803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.111778 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-kube-api-access-bxnnt" pod "b2b68edd-8617-43ae-9430-6616068a1698" (UID: "b2b68edd-8617-43ae-9430-6616068a1698"). InnerVolumeSpecName "kube-api-access-bxnnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.211302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.211259 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b68edd-8617-43ae-9430-6616068a1698-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.211302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.211300 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnnt\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-kube-api-access-bxnnt\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.213398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.213365 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b68edd-8617-43ae-9430-6616068a1698-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.213542 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.213404 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b68edd-8617-43ae-9430-6616068a1698-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.511157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.511127 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:18:26.602561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.602533 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 22:18:26.618067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618034 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-envoy\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.618186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618090 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-credential-socket\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.618186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618133 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-podinfo\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618177 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-token\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-token\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.618369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618210 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-kube-api-access-xz2wv\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-kube-api-access-xz2wv\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618255 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-data\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618288 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-socket\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618312 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-token\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-token\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618408 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-credential-socket\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618642 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618622 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f77b228e-1689-4765-bd71-1e55d0e3c498-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f77b228e-1689-4765-bd71-1e55d0e3c498-istiod-ca-cert\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.618701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618674 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-certs\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618796 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618779 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-socket\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.618858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618840 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-certs\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.618914 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618879 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istiod-ca-cert\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.618969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618914 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-kube-api-access-bhp4b\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-kube-api-access-bhp4b\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.618969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618960 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-data\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.619087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619000 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-envoy\") pod \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\" (UID: \"ff7805e9-b57f-41cf-b098-6e6e28f2dc73\") " Oct 30 22:18:26.619087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619034 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-podinfo\") pod \"f77b228e-1689-4765-bd71-1e55d0e3c498\" (UID: \"f77b228e-1689-4765-bd71-1e55d0e3c498\") " Oct 30 22:18:26.619202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618403 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-credential-socket" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.619202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.618656 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-data" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.619202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619121 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-credential-socket" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.619202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619160 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77b228e-1689-4765-bd71-1e55d0e3c498-istiod-ca-cert" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:26.619509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619481 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istiod-ca-cert" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:26.619741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619716 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.619846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619749 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.619846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619752 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-certs" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.619846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619766 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.619846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619781 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f77b228e-1689-4765-bd71-1e55d0e3c498-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.619846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.619795 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.620137 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.620001 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-certs" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.620137 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.620007 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-data" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.620255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.620201 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-socket" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.620255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.620212 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-socket" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.620982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.620957 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-token" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.622485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.622436 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-podinfo" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:26.622596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.622485 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-kube-api-access-bhp4b" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "kube-api-access-bhp4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.622596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.622537 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-podinfo" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:26.622881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.622857 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-envoy" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.622957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.622937 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-kube-api-access-xz2wv" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "kube-api-access-xz2wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.623656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.623613 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-envoy" pod "f77b228e-1689-4765-bd71-1e55d0e3c498" (UID: "f77b228e-1689-4765-bd71-1e55d0e3c498"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.625053 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.625031 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-token" pod "ff7805e9-b57f-41cf-b098-6e6e28f2dc73" (UID: "ff7805e9-b57f-41cf-b098-6e6e28f2dc73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.628460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.628441 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 22:18:26.720222 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720192 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-data\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720431 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720258 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-certs\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720431 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720295 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-token\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720431 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720358 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-kube-api-access-qw676\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-kube-api-access-qw676\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720431 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720421 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-socket\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720513 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istiod-ca-cert\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720515 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-data" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.720659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720545 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-podinfo\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720573 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-certs" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.720659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720649 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-envoy\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720690 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-credential-socket\") pod \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\" (UID: \"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1\") " Oct 30 22:18:26.720978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.720785 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-socket" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.721096 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721033 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istiod-ca-cert" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:26.721151 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721113 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-credential-socket" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.721285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721262 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721290 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721306 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721335 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721350 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721365 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721379 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721394 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721409 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhp4b\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-kube-api-access-bhp4b\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721422 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721435 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721447 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721462 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721479 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721495 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721509 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721523 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f77b228e-1689-4765-bd71-1e55d0e3c498-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.721628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.721538 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz2wv\" (UniqueName: \"kubernetes.io/projected/ff7805e9-b57f-41cf-b098-6e6e28f2dc73-kube-api-access-xz2wv\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.722689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.722668 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-podinfo" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:26.722810 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.722776 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-kube-api-access-qw676" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "kube-api-access-qw676". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.723151 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.723133 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-token" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.723505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.723481 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-envoy" pod "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" (UID: "ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.822858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.822785 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.822858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.822812 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.822858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.822821 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.822858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.822830 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw676\" (UniqueName: \"kubernetes.io/projected/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1-kube-api-access-qw676\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.988205 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.988170 4991 generic.go:356] "Generic (PLEG): container finished" podID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerID="8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7" exitCode=0 Oct 30 22:18:26.988404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.988220 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" Oct 30 22:18:26.988404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.988251 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" event={"ID":"f77b228e-1689-4765-bd71-1e55d0e3c498","Type":"ContainerDied","Data":"8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7"} Oct 30 22:18:26.988404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.988307 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn" event={"ID":"f77b228e-1689-4765-bd71-1e55d0e3c498","Type":"ContainerDied","Data":"38d103e578318c5e4ab4d36d530e316a08ae26335f44a72c8b5f54d04345517e"} Oct 30 22:18:26.988404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.988357 4991 scope.go:122] "RemoveContainer" containerID="8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7" Oct 30 22:18:26.990135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.990065 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" Oct 30 22:18:26.990135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.990102 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6" event={"ID":"b2b68edd-8617-43ae-9430-6616068a1698","Type":"ContainerDied","Data":"cc597560bb0c838c250b181dff57d49929c461f35450e2329300710f3231f2b4"} Oct 30 22:18:26.992484 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.992457 4991 generic.go:356] "Generic (PLEG): container finished" podID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerID="4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e" exitCode=0 Oct 30 22:18:26.992633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.992582 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" event={"ID":"ff7805e9-b57f-41cf-b098-6e6e28f2dc73","Type":"ContainerDied","Data":"4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e"} Oct 30 22:18:26.992633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.992620 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" event={"ID":"ff7805e9-b57f-41cf-b098-6e6e28f2dc73","Type":"ContainerDied","Data":"f7a22f081372a675b57aad130a90c5569cc26cc4a33b68328e42750e10599bd9"} Oct 30 22:18:26.992788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.992701 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z" Oct 30 22:18:26.997631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.997605 4991 generic.go:356] "Generic (PLEG): container finished" podID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerID="475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b" exitCode=0 Oct 30 22:18:26.997775 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.997641 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" event={"ID":"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1","Type":"ContainerDied","Data":"475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b"} Oct 30 22:18:26.997775 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.997675 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" event={"ID":"ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1","Type":"ContainerDied","Data":"62339a82bfbb670b1f64f754d597a84ffdf2fed0616336434d9e70f3cc195606"} Oct 30 22:18:26.997775 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:26.997739 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq" Oct 30 22:18:27.005517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.005453 4991 scope.go:122] "RemoveContainer" containerID="8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7" Oct 30 22:18:27.010515 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:27.010485 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7\": container with ID starting with 8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7 not found: ID does not exist" containerID="8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7" Oct 30 22:18:27.010656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.010527 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7"} err="failed to get container status \"8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7\": rpc error: code = NotFound desc = could not find container \"8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7\": container with ID starting with 8aba72352a37b2615d234153694c36bea16f0e604f3eb933ec70400cc8caf7d7 not found: ID does not exist" Oct 30 22:18:27.010656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.010553 4991 scope.go:122] "RemoveContainer" containerID="f21965bc286bf80fe5acbe19e4e94df7a72fbe48344788ab35466ae231554fca" Oct 30 22:18:27.020434 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.020415 4991 scope.go:122] "RemoveContainer" containerID="4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e" Oct 30 22:18:27.030997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.030978 4991 scope.go:122] "RemoveContainer" containerID="240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c" Oct 30 22:18:27.039269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.039241 4991 scope.go:122] "RemoveContainer" containerID="4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e" Oct 30 22:18:27.039564 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:27.039543 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e\": container with ID starting with 4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e not found: ID does not exist" containerID="4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e" Oct 30 22:18:27.039621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.039572 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e"} err="failed to get container status \"4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e\": rpc error: code = NotFound desc = could not find container \"4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e\": container with ID starting with 4d0eadf7741df9b8225635a037288cf2a267af9543f5c1a057d942236ecb069e not found: ID does not exist" Oct 30 22:18:27.039621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.039588 4991 scope.go:122] "RemoveContainer" containerID="240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c" Oct 30 22:18:27.039833 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:27.039817 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c\": container with ID starting with 240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c not found: ID does not exist" containerID="240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c" Oct 30 22:18:27.039874 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.039838 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c"} err="failed to get container status \"240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c\": rpc error: code = NotFound desc = could not find container \"240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c\": container with ID starting with 240bf2eb7a5f7d01c3978ebc9bfc8549a3c224bb41f3a062831cf6e7829cff2c not found: ID does not exist" Oct 30 22:18:27.039874 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.039849 4991 scope.go:122] "RemoveContainer" containerID="475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b" Oct 30 22:18:27.048033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.048018 4991 scope.go:122] "RemoveContainer" containerID="475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b" Oct 30 22:18:27.048260 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:27.048244 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b\": container with ID starting with 475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b not found: ID does not exist" containerID="475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b" Oct 30 22:18:27.048312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.048265 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b"} err="failed to get container status \"475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b\": rpc error: code = NotFound desc = could not find container \"475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b\": container with ID starting with 475081b4196b5d1003966e8ad6ea71bb189f760dd258ed56948c1306499eff7b not found: ID does not exist" Oct 30 22:18:27.148420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.148343 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq"] Oct 30 22:18:27.153160 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.153132 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-c77bq"] Oct 30 22:18:27.251511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.251473 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn"] Oct 30 22:18:27.257919 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.257890 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x2fbn"] Oct 30 22:18:27.359451 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.359421 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6"] Oct 30 22:18:27.364462 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.364433 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fpmc6"] Oct 30 22:18:27.474733 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.474702 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z"] Oct 30 22:18:27.478339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.478295 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-fj82z"] Oct 30 22:18:27.876152 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.876072 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b68edd-8617-43ae-9430-6616068a1698" path="/var/lib/kubelet/pods/b2b68edd-8617-43ae-9430-6616068a1698/volumes" Oct 30 22:18:27.876578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.876563 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" path="/var/lib/kubelet/pods/f77b228e-1689-4765-bd71-1e55d0e3c498/volumes" Oct 30 22:18:27.876971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.876958 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" path="/var/lib/kubelet/pods/ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1/volumes" Oct 30 22:18:27.877695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:27.877680 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" path="/var/lib/kubelet/pods/ff7805e9-b57f-41cf-b098-6e6e28f2dc73/volumes" Oct 30 22:18:34.967182 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967144 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7"] Oct 30 22:18:34.967665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967650 4991 cpu_manager.go:414] "containerMap: removing container" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" Oct 30 22:18:34.967713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967665 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" Oct 30 22:18:34.967713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967689 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" Oct 30 22:18:34.967713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967694 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" Oct 30 22:18:34.967713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967703 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" Oct 30 22:18:34.967713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967708 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" Oct 30 22:18:34.967873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967717 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" Oct 30 22:18:34.967873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967722 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" Oct 30 22:18:34.967873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967809 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f77b228e-1689-4765-bd71-1e55d0e3c498" containerName="istio-proxy" Oct 30 22:18:34.967873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967818 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b68edd-8617-43ae-9430-6616068a1698" containerName="istio-proxy" Oct 30 22:18:34.967873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967831 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" Oct 30 22:18:34.967873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.967837 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1857b8-ed6c-4398-a4a0-fe8bf04e2bf1" containerName="istio-proxy" Oct 30 22:18:34.973274 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.973249 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:34.993227 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.993193 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv"] Oct 30 22:18:34.993739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.993721 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" Oct 30 22:18:34.993739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.993738 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" Oct 30 22:18:34.993898 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.993837 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7805e9-b57f-41cf-b098-6e6e28f2dc73" containerName="istio-proxy" Oct 30 22:18:34.997110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:34.997090 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.015617 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.015574 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787"] Oct 30 22:18:35.019454 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.019435 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.019735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.019701 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2"] Oct 30 22:18:35.023703 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.023683 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.036740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.036720 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-twq7g\"" Oct 30 22:18:35.061956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.061934 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7"] Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.108804 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.108933 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.108964 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109025 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109058 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109087 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109115 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbcnw\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-kube-api-access-fbcnw\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109150 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109191 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109223 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109255 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109283 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109338 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109370 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rk6s\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-kube-api-access-5rk6s\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109400 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109430 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109461 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109487 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxxx\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-kube-api-access-9fxxx\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.109518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109518 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109549 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109591 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5lg\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-kube-api-access-4n5lg\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109612 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109635 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109674 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109714 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac1480-8a43-404b-9787-ed5d05d9427d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109755 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109787 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109820 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109851 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109892 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109921 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.109978 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.110002 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.110045 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.110100 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.111169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.110129 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.113411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.113388 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv"] Oct 30 22:18:35.114905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.114885 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787"] Oct 30 22:18:35.178363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.178336 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2"] Oct 30 22:18:35.210927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.210876 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxxx\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-kube-api-access-9fxxx\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.211114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.210934 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.211114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.210960 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.211114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.210994 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5lg\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-kube-api-access-4n5lg\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.211114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211018 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.211114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211041 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.211114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211086 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211121 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac1480-8a43-404b-9787-ed5d05d9427d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211157 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211185 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211217 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211243 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211274 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211300 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211366 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211569 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211778 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.211971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211866 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.213148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.212173 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.213148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.212173 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.213148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.212252 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.213148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.212978 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.213382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.213217 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac1480-8a43-404b-9787-ed5d05d9427d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.213436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.213391 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.213436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.211390 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.213541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.213485 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.213591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.213538 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.213591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.213569 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.213691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.213597 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.213691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.213641 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.214726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.214094 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.214726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.214411 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.214726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.214590 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.214726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.214691 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.215016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.214993 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.215111 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215073 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.215190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215108 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.215190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215140 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.215190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215174 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbcnw\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-kube-api-access-fbcnw\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.215449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215197 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.215449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215212 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.215449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215277 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.215449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215360 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.215449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215409 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.215449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215437 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.215881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215489 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.215881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215505 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.215881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215522 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rk6s\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-kube-api-access-5rk6s\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.215881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215573 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.215881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215605 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.215881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215656 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.215881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.215749 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.216300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.216169 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.216464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.216445 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.216564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.216482 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.216839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.216818 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.217029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.217005 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.217911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.217853 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.218642 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.218616 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.219652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.219576 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.219808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.219762 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.220453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.220425 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.223020 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.222998 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.223533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.223507 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.268751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.268726 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxxx\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-kube-api-access-9fxxx\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.268907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.268886 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5lg\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-kube-api-access-4n5lg\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.289636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.289611 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.293629 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.293608 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-f84b2\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.293766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.293744 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbcnw\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-kube-api-access-fbcnw\") pod \"maas-default-gateway-openshift-default-6f54779779-gz787\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.293840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.293820 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rk6s\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-kube-api-access-5rk6s\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.294883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.294861 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-7hsfv\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.305193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.305171 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-sxjs7\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.312637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.312608 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:35.333442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.333420 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:35.340805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.340664 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:35.587713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.587686 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:35.744047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.744014 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv"] Oct 30 22:18:35.750162 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:18:35.750136 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ac1480_8a43_404b_9787_ed5d05d9427d.slice/crio-0de5bfbaffd9ad7d17e27edf7fe2b543f673b667b488271d8896c05679b46efb WatchSource:0}: Error finding container 0de5bfbaffd9ad7d17e27edf7fe2b543f673b667b488271d8896c05679b46efb: Status 404 returned error can't find the container with id 0de5bfbaffd9ad7d17e27edf7fe2b543f673b667b488271d8896c05679b46efb Oct 30 22:18:35.753180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.752645 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.753180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.752727 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.753180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.752766 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.757117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.757093 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787"] Oct 30 22:18:35.768098 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:18:35.768067 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cf0461_d0c9_4bc7_8c21_4572cb9fc95e.slice/crio-085158baadc669ab7a94173d74b640b6c2c639de2c2298faad213babd33a2c47 WatchSource:0}: Error finding container 085158baadc669ab7a94173d74b640b6c2c639de2c2298faad213babd33a2c47: Status 404 returned error can't find the container with id 085158baadc669ab7a94173d74b640b6c2c639de2c2298faad213babd33a2c47 Oct 30 22:18:35.770978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.770953 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.771025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.771008 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.771064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.771040 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.787780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.787755 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2"] Oct 30 22:18:35.792100 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:18:35.792072 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b2c50f_940b_4d9c_9c27_4f83b7667a78.slice/crio-f638f6c8d961870627701b796435bb34ba649a7418806b75dadf3e9a16ec5a0e WatchSource:0}: Error finding container f638f6c8d961870627701b796435bb34ba649a7418806b75dadf3e9a16ec5a0e: Status 404 returned error can't find the container with id f638f6c8d961870627701b796435bb34ba649a7418806b75dadf3e9a16ec5a0e Oct 30 22:18:35.795527 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.795489 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.795605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.795564 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.795605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.795595 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.971914 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.971877 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7"] Oct 30 22:18:35.976705 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:18:35.976675 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe4e4a8_11eb_4720_bfac_a62d1136d8bd.slice/crio-07d304b3171b68cea9fc8845bf3a190d9d5dc5b1180eae2ddd8e4a2f6f0cecf8 WatchSource:0}: Error finding container 07d304b3171b68cea9fc8845bf3a190d9d5dc5b1180eae2ddd8e4a2f6f0cecf8: Status 404 returned error can't find the container with id 07d304b3171b68cea9fc8845bf3a190d9d5dc5b1180eae2ddd8e4a2f6f0cecf8 Oct 30 22:18:35.979955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.979922 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.980031 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.979996 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:35.980031 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:35.980027 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:18:36.063947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.063905 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" event={"ID":"31b2c50f-940b-4d9c-9c27-4f83b7667a78","Type":"ContainerStarted","Data":"cd5c8217de1bf3fd6c6d98ad40594c5256ef5ed0458226e51c7f9736ba88aeac"} Oct 30 22:18:36.064091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.063979 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" event={"ID":"31b2c50f-940b-4d9c-9c27-4f83b7667a78","Type":"ContainerStarted","Data":"f638f6c8d961870627701b796435bb34ba649a7418806b75dadf3e9a16ec5a0e"} Oct 30 22:18:36.067762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.067075 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" event={"ID":"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd","Type":"ContainerStarted","Data":"bb15d84f712e6196d9a3d460d326a91ac9866989a5d346cd4950a9705248e4a1"} Oct 30 22:18:36.067762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.067125 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" event={"ID":"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd","Type":"ContainerStarted","Data":"07d304b3171b68cea9fc8845bf3a190d9d5dc5b1180eae2ddd8e4a2f6f0cecf8"} Oct 30 22:18:36.069749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.069724 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" event={"ID":"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e","Type":"ContainerStarted","Data":"9566873e46f70d9bae1a8cd977183d22dc753ede94d28f6828d742e442214e1d"} Oct 30 22:18:36.069870 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.069777 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" event={"ID":"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e","Type":"ContainerStarted","Data":"085158baadc669ab7a94173d74b640b6c2c639de2c2298faad213babd33a2c47"} Oct 30 22:18:36.077432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.077405 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" event={"ID":"37ac1480-8a43-404b-9787-ed5d05d9427d","Type":"ContainerStarted","Data":"fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38"} Oct 30 22:18:36.077538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.077457 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" event={"ID":"37ac1480-8a43-404b-9787-ed5d05d9427d","Type":"ContainerStarted","Data":"0de5bfbaffd9ad7d17e27edf7fe2b543f673b667b488271d8896c05679b46efb"} Oct 30 22:18:36.182043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.181934 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podStartSLOduration=2.181916769 podStartE2EDuration="2.181916769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.178781791 +0000 UTC m=+1936.478191672" watchObservedRunningTime="2026-10-30 22:18:36.181916769 +0000 UTC m=+1936.481326649" Oct 30 22:18:36.300669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.300608 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podStartSLOduration=2.300590713 podStartE2EDuration="2.300590713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.296299199 +0000 UTC m=+1936.595709094" watchObservedRunningTime="2026-10-30 22:18:36.300590713 +0000 UTC m=+1936.600000594" Oct 30 22:18:36.316616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.316585 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:36.321456 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.321426 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:36.321661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.321637 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:36.338409 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.338381 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:36.340504 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.340478 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:36.340631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.340528 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:36.341633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.341617 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:36.342849 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.342825 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:36.342946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.342873 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:36.413895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.413837 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podStartSLOduration=2.413819487 podStartE2EDuration="2.413819487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.411679795 +0000 UTC m=+1936.711089679" watchObservedRunningTime="2026-10-30 22:18:36.413819487 +0000 UTC m=+1936.713229366" Oct 30 22:18:36.610374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:36.609818 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:37.317593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.317557 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:37.318181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.318151 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:37.337343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.334661 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:37.337343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.334720 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:37.346341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.342668 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:37.346341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.342724 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:37.593237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.593141 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:37.593508 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:37.593477 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:38.314341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.313383 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:38.314341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.313467 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:38.338068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.338038 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:38.343142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.343112 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:38.343306 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.343048 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:38.343461 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.343440 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:38.594407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.588717 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:38.599442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:38.599406 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:39.446344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.442679 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:39.446344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.442770 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:39.446344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.443160 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:39.446344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.443185 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:39.446344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.443243 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:39.446344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.443263 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:39.597342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.596708 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:39.597342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:39.596785 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:40.319966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.319435 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:40.319966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.319515 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:40.348770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.348732 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:40.349012 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.348987 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:40.350188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.350164 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:40.350345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.350306 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:40.590853 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.590758 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:40.591440 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:40.591404 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:41.326759 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.325447 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:41.326759 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.325511 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:41.334257 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.334150 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:41.334257 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.334204 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:41.341764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.341736 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:41.341981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.341958 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:41.592896 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.592717 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:41.592896 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:41.592780 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:42.075344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.074470 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:18:42.189462 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.189417 4991 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:42.313351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.312919 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:42.313351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.312980 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:42.335794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.335704 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:42.336046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.336019 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:42.343343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.335502 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:42.343873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.343778 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:42.343873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.343820 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:42.589394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.588827 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:42.633574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.633514 4991 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 30 22:18:42.634099 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:42.633735 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:43.313122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.313074 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:43.313419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.313377 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:43.333968 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.333846 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:43.333968 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.333905 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:43.341533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.341495 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:43.341696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.341570 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:43.588779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.588692 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:43.588779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:43.588758 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:44.313995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.313958 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:18:44.314514 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.314022 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 22:18:44.334665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.334619 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:18:44.334909 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.334883 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 22:18:44.341337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.341295 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:18:44.341444 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.341368 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:18:44.588515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.588415 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-sxjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:18:44.588515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:44.588476 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 22:18:46.358404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.358365 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-f84b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:46.359004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.358974 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:46.367544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.367488 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-7hsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:46.367785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.367747 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:46.390540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.390483 4991 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gz787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:46.390785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.390739 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:46.439220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.439171 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:46.440001 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.439987 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:46.441233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.441213 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:18:46.506654 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.506607 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:46.507070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.507048 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:46.508782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.508759 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:18:46.540594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.540561 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:46.541121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.541070 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:46.542247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.542206 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:18:46.583344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.580659 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podStartSLOduration=12.580641694 podStartE2EDuration="12.580641694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.537615831 +0000 UTC m=+1936.837025711" watchObservedRunningTime="2026-10-30 22:18:46.580641694 +0000 UTC m=+1946.880051584" Oct 30 22:18:46.604903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.604870 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:46.605183 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.605163 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:46.606034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:18:46.606012 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:18:47.909260 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:18:47.909216 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:19:17.911494 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:17.911452 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:19:19.946934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:19.946902 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw"] Oct 30 22:19:19.947457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:19.947105 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" podUID="c8e8c5f9-bdc6-480d-afb5-af949b21b993" containerName="payload-pre-processing" containerID="cri-o://e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75" gracePeriod=30 Oct 30 22:19:20.232735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.232705 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:19:20.304495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.304464 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e8c5f9-bdc6-480d-afb5-af949b21b993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e8c5f9-bdc6-480d-afb5-af949b21b993-plugins-config-volume\") pod \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " Oct 30 22:19:20.304639 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.304501 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e8c5f9-bdc6-480d-afb5-af949b21b993-kube-api-access-qvf4j\" (UniqueName: \"kubernetes.io/projected/c8e8c5f9-bdc6-480d-afb5-af949b21b993-kube-api-access-qvf4j\") pod \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\" (UID: \"c8e8c5f9-bdc6-480d-afb5-af949b21b993\") " Oct 30 22:19:20.304813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.304787 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e8c5f9-bdc6-480d-afb5-af949b21b993-plugins-config-volume" pod "c8e8c5f9-bdc6-480d-afb5-af949b21b993" (UID: "c8e8c5f9-bdc6-480d-afb5-af949b21b993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:20.304937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.304923 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e8c5f9-bdc6-480d-afb5-af949b21b993-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.306708 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.306685 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e8c5f9-bdc6-480d-afb5-af949b21b993-kube-api-access-qvf4j" pod "c8e8c5f9-bdc6-480d-afb5-af949b21b993" (UID: "c8e8c5f9-bdc6-480d-afb5-af949b21b993"). InnerVolumeSpecName "kube-api-access-qvf4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:20.419408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.419366 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvf4j\" (UniqueName: \"kubernetes.io/projected/c8e8c5f9-bdc6-480d-afb5-af949b21b993-kube-api-access-qvf4j\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.690825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.690782 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk"] Oct 30 22:19:20.691040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.691000 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" podUID="a3b68d4d-02e4-43a2-89b5-6d5ef9672446" containerName="maas-api" containerID="cri-o://e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3" gracePeriod=30 Oct 30 22:19:20.998261 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:20.998234 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:19:21.015382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.015354 4991 generic.go:356] "Generic (PLEG): container finished" podID="a3b68d4d-02e4-43a2-89b5-6d5ef9672446" containerID="e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3" exitCode=0 Oct 30 22:19:21.015517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.015398 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" Oct 30 22:19:21.015517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.015441 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" event={"ID":"a3b68d4d-02e4-43a2-89b5-6d5ef9672446","Type":"ContainerDied","Data":"e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3"} Oct 30 22:19:21.015517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.015500 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk" event={"ID":"a3b68d4d-02e4-43a2-89b5-6d5ef9672446","Type":"ContainerDied","Data":"5b2d655028e74863e39678e55a99b5143ac8e9b17b2dda0f324bf31abed00765"} Oct 30 22:19:21.015687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.015532 4991 scope.go:122] "RemoveContainer" containerID="e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3" Oct 30 22:19:21.016663 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.016643 4991 generic.go:356] "Generic (PLEG): container finished" podID="c8e8c5f9-bdc6-480d-afb5-af949b21b993" containerID="e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75" exitCode=0 Oct 30 22:19:21.016758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.016697 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" Oct 30 22:19:21.016758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.016720 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" event={"ID":"c8e8c5f9-bdc6-480d-afb5-af949b21b993","Type":"ContainerDied","Data":"e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75"} Oct 30 22:19:21.016758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.016757 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw" event={"ID":"c8e8c5f9-bdc6-480d-afb5-af949b21b993","Type":"ContainerDied","Data":"8205963be698b5be54316379db3876939e8f2c7cb8ac7a0a8832a84a88ff73c9"} Oct 30 22:19:21.026665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.026562 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-metrics-certs\") pod \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " Oct 30 22:19:21.026735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.026689 4991 scope.go:122] "RemoveContainer" containerID="e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3" Oct 30 22:19:21.026953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.026933 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-kube-api-access-5lmzk\" (UniqueName: \"kubernetes.io/projected/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-kube-api-access-5lmzk\") pod \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " Oct 30 22:19:21.027069 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:21.026960 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3\": container with ID starting with e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3 not found: ID does not exist" containerID="e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3" Oct 30 22:19:21.027069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.026993 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls\") pod \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " Oct 30 22:19:21.027199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.026990 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3"} err="failed to get container status \"e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3\": rpc error: code = NotFound desc = could not find container \"e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3\": container with ID starting with e728bf96ca3bd50734146fe503b8094e2d4161460fe1d20bba569665333aecd3 not found: ID does not exist" Oct 30 22:19:21.027199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.027150 4991 scope.go:122] "RemoveContainer" containerID="e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75" Oct 30 22:19:21.027273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.027125 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-openshift-service-ca\") pod \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\" (UID: \"a3b68d4d-02e4-43a2-89b5-6d5ef9672446\") " Oct 30 22:19:21.027838 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.027817 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-openshift-service-ca" pod "a3b68d4d-02e4-43a2-89b5-6d5ef9672446" (UID: "a3b68d4d-02e4-43a2-89b5-6d5ef9672446"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:21.028959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.028939 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-metrics-certs" pod "a3b68d4d-02e4-43a2-89b5-6d5ef9672446" (UID: "a3b68d4d-02e4-43a2-89b5-6d5ef9672446"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:21.029811 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.029777 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-kube-api-access-5lmzk" pod "a3b68d4d-02e4-43a2-89b5-6d5ef9672446" (UID: "a3b68d4d-02e4-43a2-89b5-6d5ef9672446"). InnerVolumeSpecName "kube-api-access-5lmzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:21.030095 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.030067 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls" pod "a3b68d4d-02e4-43a2-89b5-6d5ef9672446" (UID: "a3b68d4d-02e4-43a2-89b5-6d5ef9672446"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:21.078132 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.078104 4991 scope.go:122] "RemoveContainer" containerID="e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75" Oct 30 22:19:21.078576 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:21.078546 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75\": container with ID starting with e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75 not found: ID does not exist" containerID="e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75" Oct 30 22:19:21.078697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.078587 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75"} err="failed to get container status \"e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75\": rpc error: code = NotFound desc = could not find container \"e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75\": container with ID starting with e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75 not found: ID does not exist" Oct 30 22:19:21.128799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.128760 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:21.128799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.128793 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:21.128981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.128808 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:21.128981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.128823 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmzk\" (UniqueName: \"kubernetes.io/projected/a3b68d4d-02e4-43a2-89b5-6d5ef9672446-kube-api-access-5lmzk\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:21.216736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.216701 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz"] Oct 30 22:19:21.216952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.216925 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" podUID="28db1479-29e2-4366-b1a0-96f85c9957da" containerName="payload-pre-processing" containerID="cri-o://91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5" gracePeriod=30 Oct 30 22:19:21.231056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.231028 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw"] Oct 30 22:19:21.233852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.233831 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-5d89dfb87-dkvnw"] Oct 30 22:19:21.500895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.500859 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk"] Oct 30 22:19:21.503703 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.503679 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:19:21.504557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.504532 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04d404-554bdbd886-qxbxk"] Oct 30 22:19:21.534721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.534696 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28db1479-29e2-4366-b1a0-96f85c9957da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db1479-29e2-4366-b1a0-96f85c9957da-plugins-config-volume\") pod \"28db1479-29e2-4366-b1a0-96f85c9957da\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " Oct 30 22:19:21.534827 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.534762 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28db1479-29e2-4366-b1a0-96f85c9957da-kube-api-access-wdd7l\" (UniqueName: \"kubernetes.io/projected/28db1479-29e2-4366-b1a0-96f85c9957da-kube-api-access-wdd7l\") pod \"28db1479-29e2-4366-b1a0-96f85c9957da\" (UID: \"28db1479-29e2-4366-b1a0-96f85c9957da\") " Oct 30 22:19:21.535073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.535051 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28db1479-29e2-4366-b1a0-96f85c9957da-plugins-config-volume" pod "28db1479-29e2-4366-b1a0-96f85c9957da" (UID: "28db1479-29e2-4366-b1a0-96f85c9957da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:21.536835 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.536807 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28db1479-29e2-4366-b1a0-96f85c9957da-kube-api-access-wdd7l" pod "28db1479-29e2-4366-b1a0-96f85c9957da" (UID: "28db1479-29e2-4366-b1a0-96f85c9957da"). InnerVolumeSpecName "kube-api-access-wdd7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:21.636212 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.636179 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db1479-29e2-4366-b1a0-96f85c9957da-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:21.636212 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.636209 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdd7l\" (UniqueName: \"kubernetes.io/projected/28db1479-29e2-4366-b1a0-96f85c9957da-kube-api-access-wdd7l\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:21.792885 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:21.792784 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:21.875166 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.875133 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b68d4d-02e4-43a2-89b5-6d5ef9672446" path="/var/lib/kubelet/pods/a3b68d4d-02e4-43a2-89b5-6d5ef9672446/volumes" Oct 30 22:19:21.875587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:21.875572 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e8c5f9-bdc6-480d-afb5-af949b21b993" path="/var/lib/kubelet/pods/c8e8c5f9-bdc6-480d-afb5-af949b21b993/volumes" Oct 30 22:19:22.026114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.026086 4991 generic.go:356] "Generic (PLEG): container finished" podID="28db1479-29e2-4366-b1a0-96f85c9957da" containerID="91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5" exitCode=0 Oct 30 22:19:22.026574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.026168 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" Oct 30 22:19:22.026574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.026174 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" event={"ID":"28db1479-29e2-4366-b1a0-96f85c9957da","Type":"ContainerDied","Data":"91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5"} Oct 30 22:19:22.026574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.026273 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz" event={"ID":"28db1479-29e2-4366-b1a0-96f85c9957da","Type":"ContainerDied","Data":"9518b8b77449e59167b0473e499e8076cf86574a28b485a41e9fa10edd0a9d63"} Oct 30 22:19:22.026574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.026307 4991 scope.go:122] "RemoveContainer" containerID="91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5" Oct 30 22:19:22.038289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.038266 4991 scope.go:122] "RemoveContainer" containerID="91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5" Oct 30 22:19:22.038644 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:22.038617 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5\": container with ID starting with 91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5 not found: ID does not exist" containerID="91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5" Oct 30 22:19:22.038733 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.038659 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5"} err="failed to get container status \"91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5\": rpc error: code = NotFound desc = could not find container \"91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5\": container with ID starting with 91577487dc5dcf1b979cb1e4194b519830c99b2c9c6e795aecd468c4b9711dc5 not found: ID does not exist" Oct 30 22:19:22.187487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.187409 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz"] Oct 30 22:19:22.193069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:22.193042 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-7d8b8778bc-ddrlz"] Oct 30 22:19:23.881342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:23.880931 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28db1479-29e2-4366-b1a0-96f85c9957da" path="/var/lib/kubelet/pods/28db1479-29e2-4366-b1a0-96f85c9957da/volumes" Oct 30 22:19:28.264464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.264433 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b"] Oct 30 22:19:28.265091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265074 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c8e8c5f9-bdc6-480d-afb5-af949b21b993" containerName="payload-pre-processing" Oct 30 22:19:28.265158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265093 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e8c5f9-bdc6-480d-afb5-af949b21b993" containerName="payload-pre-processing" Oct 30 22:19:28.265158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265106 4991 cpu_manager.go:414] "containerMap: removing container" podUID="a3b68d4d-02e4-43a2-89b5-6d5ef9672446" containerName="maas-api" Oct 30 22:19:28.265158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265112 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b68d4d-02e4-43a2-89b5-6d5ef9672446" containerName="maas-api" Oct 30 22:19:28.265158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265123 4991 cpu_manager.go:414] "containerMap: removing container" podUID="28db1479-29e2-4366-b1a0-96f85c9957da" containerName="payload-pre-processing" Oct 30 22:19:28.265158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265132 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28db1479-29e2-4366-b1a0-96f85c9957da" containerName="payload-pre-processing" Oct 30 22:19:28.265416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265232 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e8c5f9-bdc6-480d-afb5-af949b21b993" containerName="payload-pre-processing" Oct 30 22:19:28.265416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265244 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b68d4d-02e4-43a2-89b5-6d5ef9672446" containerName="maas-api" Oct 30 22:19:28.265416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.265255 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="28db1479-29e2-4366-b1a0-96f85c9957da" containerName="payload-pre-processing" Oct 30 22:19:28.270453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.270428 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.297123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.297100 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.297223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.297129 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqrj\" (UniqueName: \"kubernetes.io/projected/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-kube-api-access-7nqrj\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.320169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.320148 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4576d9\"" Oct 30 22:19:28.366688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.366661 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4576d9-dockercfg-ht95n\"" Oct 30 22:19:28.370229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.370203 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b"] Oct 30 22:19:28.398052 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.398027 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.398152 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.398060 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqrj\" (UniqueName: \"kubernetes.io/projected/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-kube-api-access-7nqrj\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.398742 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.398721 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.456118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.456082 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqrj\" (UniqueName: \"kubernetes.io/projected/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-kube-api-access-7nqrj\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.583763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.583687 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:28.964224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:28.964144 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b"] Oct 30 22:19:28.971001 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:19:28.970973 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22627eb_ff59_47c4_b21d_d49ff8e8c4fc.slice/crio-efec69836718682e289958eb72b7b74d3b4346d4316b2267f9587ed8c202b7a6 WatchSource:0}: Error finding container efec69836718682e289958eb72b7b74d3b4346d4316b2267f9587ed8c202b7a6: Status 404 returned error can't find the container with id efec69836718682e289958eb72b7b74d3b4346d4316b2267f9587ed8c202b7a6 Oct 30 22:19:29.083181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:29.083092 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" event={"ID":"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc","Type":"ContainerStarted","Data":"0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a"} Oct 30 22:19:29.083181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:29.083161 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" event={"ID":"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc","Type":"ContainerStarted","Data":"efec69836718682e289958eb72b7b74d3b4346d4316b2267f9587ed8c202b7a6"} Oct 30 22:19:29.084113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:29.084089 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:29.197158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:29.197106 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" podStartSLOduration=1.197093585 podStartE2EDuration="1.197093585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:29.196520849 +0000 UTC m=+1989.495930748" watchObservedRunningTime="2026-10-30 22:19:29.197093585 +0000 UTC m=+1989.496503464" Oct 30 22:19:29.254651 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:29.254617 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g"] Oct 30 22:19:29.254874 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:29.254835 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" containerID="cri-o://ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696" gracePeriod=30 Oct 30 22:19:31.873341 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:31.870810 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:34.489509 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:34.489465 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:34.533814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.533761 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:19:34.619956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.619929 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9384c5fc-2208-42b0-b77c-bab283096d61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384c5fc-2208-42b0-b77c-bab283096d61-istiod-ca-cert\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620012 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-istio-token\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-istio-token\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620047 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-credential-socket\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620104 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-envoy\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620146 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-kube-api-access-9hv4t\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620213 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-certs\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620289 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9384c5fc-2208-42b0-b77c-bab283096d61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384c5fc-2208-42b0-b77c-bab283096d61-istio-podinfo\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620350 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9384c5fc-2208-42b0-b77c-bab283096d61-istiod-ca-cert" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:34.620473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620413 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-data\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620453 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-socket\") pod \"9384c5fc-2208-42b0-b77c-bab283096d61\" (UID: \"9384c5fc-2208-42b0-b77c-bab283096d61\") " Oct 30 22:19:34.620868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620528 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-certs" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:34.620868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620663 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-data" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:34.620868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620798 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-credential-socket" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:34.620868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620810 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-socket" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:34.621176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.620984 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.621176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.621006 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.621176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.621022 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384c5fc-2208-42b0-b77c-bab283096d61-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.621176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.621039 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.621176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.621053 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.622477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.622453 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-kube-api-access-9hv4t" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "kube-api-access-9hv4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:34.622575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.622549 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-istio-token" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:34.622917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.622893 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9384c5fc-2208-42b0-b77c-bab283096d61-istio-podinfo" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:34.623104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.623080 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-envoy" pod "9384c5fc-2208-42b0-b77c-bab283096d61" (UID: "9384c5fc-2208-42b0-b77c-bab283096d61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:34.722475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.722443 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384c5fc-2208-42b0-b77c-bab283096d61-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.722475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.722471 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-kube-api-access-9hv4t\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.722475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.722482 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384c5fc-2208-42b0-b77c-bab283096d61-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:34.722747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:34.722492 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384c5fc-2208-42b0-b77c-bab283096d61-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:35.130568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.130533 4991 generic.go:356] "Generic (PLEG): container finished" podID="9384c5fc-2208-42b0-b77c-bab283096d61" containerID="ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696" exitCode=0 Oct 30 22:19:35.130753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.130599 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" Oct 30 22:19:35.130753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.130620 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" event={"ID":"9384c5fc-2208-42b0-b77c-bab283096d61","Type":"ContainerDied","Data":"ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696"} Oct 30 22:19:35.130753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.130688 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g" event={"ID":"9384c5fc-2208-42b0-b77c-bab283096d61","Type":"ContainerDied","Data":"2f602b500aaff542fc505979776060465fff2e69673446bd780f6c4a692a2dac"} Oct 30 22:19:35.130753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.130724 4991 scope.go:122] "RemoveContainer" containerID="ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696" Oct 30 22:19:35.141535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.141515 4991 scope.go:122] "RemoveContainer" containerID="f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a" Oct 30 22:19:35.153370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.153350 4991 scope.go:122] "RemoveContainer" containerID="ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696" Oct 30 22:19:35.153663 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:35.153640 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696\": container with ID starting with ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696 not found: ID does not exist" containerID="ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696" Oct 30 22:19:35.153714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.153679 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696"} err="failed to get container status \"ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696\": rpc error: code = NotFound desc = could not find container \"ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696\": container with ID starting with ad9275d2b47435dd8b50a91d8131468d81f0f547862d7dffced21e1158b56696 not found: ID does not exist" Oct 30 22:19:35.153714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.153702 4991 scope.go:122] "RemoveContainer" containerID="f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a" Oct 30 22:19:35.154019 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:35.153996 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a\": container with ID starting with f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a not found: ID does not exist" containerID="f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a" Oct 30 22:19:35.154124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.154028 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a"} err="failed to get container status \"f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a\": rpc error: code = NotFound desc = could not find container \"f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a\": container with ID starting with f9a44ebb51d36bc5a4e4bfdf274f68406011f4ccdab5b3e4f484d3fbe786a70a not found: ID does not exist" Oct 30 22:19:35.287515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.287481 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g"] Oct 30 22:19:35.334587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.334557 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad7da0-openshift-default-85bcfdbc69-gbn9g"] Oct 30 22:19:35.876986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:35.876945 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" path="/var/lib/kubelet/pods/9384c5fc-2208-42b0-b77c-bab283096d61/volumes" Oct 30 22:19:41.099442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:19:41.099410 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:19:41.950595 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:41.950563 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:47.910785 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:47.910748 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:19:49.547333 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:49.547279 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:51.963379 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:19:51.963341 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:20:02.048053 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:02.047966 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:20:04.474063 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:04.474014 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:20:07.241534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.241497 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq"] Oct 30 22:20:07.243186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.243154 4991 cpu_manager.go:414] "containerMap: removing container" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" Oct 30 22:20:07.243395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.243378 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" Oct 30 22:20:07.244024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.244007 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" Oct 30 22:20:07.244175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.244161 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" Oct 30 22:20:07.872343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.871676 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq"] Oct 30 22:20:07.880140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.878483 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.934299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.934264 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-podinfo\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.934637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.934615 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rdt\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-kube-api-access-67rdt\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.935194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.935174 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-token\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.935576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.935557 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-credential-socket\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.935793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.935775 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-envoy\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.943467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.943445 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-certs\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.943690 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.943671 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istiod-ca-cert\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.943999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.943981 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-socket\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:07.944165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:07.944147 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-data\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.005695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.005660 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6a3357-openshift-default-dockercfg-nb7jw\"" Oct 30 22:20:08.046103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.046064 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-podinfo\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.067606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.067439 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:20:08.069149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.069117 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rdt\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-kube-api-access-67rdt\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.069802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.069779 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-token\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.070915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.070874 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-credential-socket\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.077686 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.077544 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-credential-socket\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.097421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.078050 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-envoy\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.097628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.097609 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-certs\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.098532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.098510 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istiod-ca-cert\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.099474 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.099455 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-socket\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.100027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.100008 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-data\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.100637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.098041 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-certs\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.100771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.099879 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-socket\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.100865 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.099299 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istiod-ca-cert\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.100954 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.100471 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-data\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.121117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.121095 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-envoy\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.135027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.134964 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-podinfo\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.175355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.170048 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rdt\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-kube-api-access-67rdt\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.185341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.181787 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-token\") pod \"e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:08.250470 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:08.250443 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:09.115447 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:09.115350 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq"] Oct 30 22:20:09.121388 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:20:09.121211 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0e72d9_7a32_4f56_b382_c998e1bf046f.slice/crio-7ea11a93e94566a8a1c572397558456515fbfe38f8909f108bbcd2646279e0f3 WatchSource:0}: Error finding container 7ea11a93e94566a8a1c572397558456515fbfe38f8909f108bbcd2646279e0f3: Status 404 returned error can't find the container with id 7ea11a93e94566a8a1c572397558456515fbfe38f8909f108bbcd2646279e0f3 Oct 30 22:20:09.129869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:09.128491 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:09.129869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:09.128578 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:09.129869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:09.128618 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:09.932344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:09.931461 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" event={"ID":"0c0e72d9-7a32-4f56-b382-c998e1bf046f","Type":"ContainerStarted","Data":"426319d52a16627dfa1ba4fa21727cb49d8f386e3dac7537b6c50bd756761994"} Oct 30 22:20:09.932344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:09.931532 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" event={"ID":"0c0e72d9-7a32-4f56-b382-c998e1bf046f","Type":"ContainerStarted","Data":"7ea11a93e94566a8a1c572397558456515fbfe38f8909f108bbcd2646279e0f3"} Oct 30 22:20:10.095681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:10.095628 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" podStartSLOduration=3.095613731 podStartE2EDuration="3.095613731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:10.092139961 +0000 UTC m=+2030.391549843" watchObservedRunningTime="2026-10-30 22:20:10.095613731 +0000 UTC m=+2030.395023613" Oct 30 22:20:10.251853 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:10.251817 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:10.281701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:10.272891 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:10.937061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:10.937034 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:10.937854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:10.937832 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:20:12.062350 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:12.062305 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:20:15.764769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.764733 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj"] Oct 30 22:20:15.765627 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.765604 4991 cpu_manager.go:414] "containerMap: removing container" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" Oct 30 22:20:15.765722 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.765630 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9384c5fc-2208-42b0-b77c-bab283096d61" containerName="istio-proxy" Oct 30 22:20:15.769903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.769884 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:15.823615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.822933 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj"] Oct 30 22:20:15.865076 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.865047 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8daffd-dockercfg-z8sv8\"" Oct 30 22:20:15.865234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.865095 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8daffd\"" Oct 30 22:20:15.873763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.873739 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2wx\" (UniqueName: \"kubernetes.io/projected/7caf6588-e2a5-4f42-b401-2810e464570a-kube-api-access-wr2wx\") pod \"payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:15.873902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.873805 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7caf6588-e2a5-4f42-b401-2810e464570a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:15.877705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.877674 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng"] Oct 30 22:20:15.880963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.877875 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" podUID="b87c71bf-75f8-4a27-91cb-1547f7629af6" containerName="maas-api" containerID="cri-o://fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614" gracePeriod=30 Oct 30 22:20:15.975159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.975124 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2wx\" (UniqueName: \"kubernetes.io/projected/7caf6588-e2a5-4f42-b401-2810e464570a-kube-api-access-wr2wx\") pod \"payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:15.975353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.975211 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7caf6588-e2a5-4f42-b401-2810e464570a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:15.975964 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:15.975940 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7caf6588-e2a5-4f42-b401-2810e464570a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:16.034407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.034294 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2wx\" (UniqueName: \"kubernetes.io/projected/7caf6588-e2a5-4f42-b401-2810e464570a-kube-api-access-wr2wx\") pod \"payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:16.083917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.083682 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:16.183173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.183141 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:20:16.280923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.280879 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-metrics-certs\") pod \"b87c71bf-75f8-4a27-91cb-1547f7629af6\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " Oct 30 22:20:16.281119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.281016 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls\") pod \"b87c71bf-75f8-4a27-91cb-1547f7629af6\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " Oct 30 22:20:16.281119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.281058 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87c71bf-75f8-4a27-91cb-1547f7629af6-kube-api-access-jvqt7\" (UniqueName: \"kubernetes.io/projected/b87c71bf-75f8-4a27-91cb-1547f7629af6-kube-api-access-jvqt7\") pod \"b87c71bf-75f8-4a27-91cb-1547f7629af6\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " Oct 30 22:20:16.281240 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.281153 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87c71bf-75f8-4a27-91cb-1547f7629af6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87c71bf-75f8-4a27-91cb-1547f7629af6-openshift-service-ca\") pod \"b87c71bf-75f8-4a27-91cb-1547f7629af6\" (UID: \"b87c71bf-75f8-4a27-91cb-1547f7629af6\") " Oct 30 22:20:16.282022 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.281991 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87c71bf-75f8-4a27-91cb-1547f7629af6-openshift-service-ca" pod "b87c71bf-75f8-4a27-91cb-1547f7629af6" (UID: "b87c71bf-75f8-4a27-91cb-1547f7629af6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:16.285087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.285016 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls" pod "b87c71bf-75f8-4a27-91cb-1547f7629af6" (UID: "b87c71bf-75f8-4a27-91cb-1547f7629af6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:16.285087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.285067 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87c71bf-75f8-4a27-91cb-1547f7629af6-kube-api-access-jvqt7" pod "b87c71bf-75f8-4a27-91cb-1547f7629af6" (UID: "b87c71bf-75f8-4a27-91cb-1547f7629af6"). InnerVolumeSpecName "kube-api-access-jvqt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:16.287279 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.287247 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-metrics-certs" pod "b87c71bf-75f8-4a27-91cb-1547f7629af6" (UID: "b87c71bf-75f8-4a27-91cb-1547f7629af6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:16.382521 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.382487 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87c71bf-75f8-4a27-91cb-1547f7629af6-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.382521 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.382518 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.382726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.382535 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87c71bf-75f8-4a27-91cb-1547f7629af6-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.382726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.382547 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqt7\" (UniqueName: \"kubernetes.io/projected/b87c71bf-75f8-4a27-91cb-1547f7629af6-kube-api-access-jvqt7\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.413217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.413184 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b"] Oct 30 22:20:16.413455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.413430 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" podUID="b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" containerName="payload-pre-processing" containerID="cri-o://0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a" gracePeriod=30 Oct 30 22:20:16.514552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.514521 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj"] Oct 30 22:20:16.523644 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:20:16.523612 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caf6588_e2a5_4f42_b401_2810e464570a.slice/crio-3eb7a3a18459b33e06b181a102126e208491f9b8b285bde340ecd1251fcc7f1f WatchSource:0}: Error finding container 3eb7a3a18459b33e06b181a102126e208491f9b8b285bde340ecd1251fcc7f1f: Status 404 returned error can't find the container with id 3eb7a3a18459b33e06b181a102126e208491f9b8b285bde340ecd1251fcc7f1f Oct 30 22:20:16.681491 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.681466 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:20:16.786299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.786197 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-plugins-config-volume\") pod \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " Oct 30 22:20:16.786299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.786281 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-kube-api-access-7nqrj\" (UniqueName: \"kubernetes.io/projected/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-kube-api-access-7nqrj\") pod \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\" (UID: \"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc\") " Oct 30 22:20:16.786886 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.786663 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-plugins-config-volume" pod "b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" (UID: "b22627eb-ff59-47c4-b21d-d49ff8e8c4fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:16.786886 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.786787 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.788637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.788609 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-kube-api-access-7nqrj" pod "b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" (UID: "b22627eb-ff59-47c4-b21d-d49ff8e8c4fc"). InnerVolumeSpecName "kube-api-access-7nqrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:16.803000 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:16.802972 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:20:16.803131 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:16.803067 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:20:16.896343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.893206 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nqrj\" (UniqueName: \"kubernetes.io/projected/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc-kube-api-access-7nqrj\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.999133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.999098 4991 generic.go:356] "Generic (PLEG): container finished" podID="b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" containerID="0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a" exitCode=0 Oct 30 22:20:16.999381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.999168 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" Oct 30 22:20:16.999381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.999181 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" event={"ID":"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc","Type":"ContainerDied","Data":"0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a"} Oct 30 22:20:16.999381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.999267 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b" event={"ID":"b22627eb-ff59-47c4-b21d-d49ff8e8c4fc","Type":"ContainerDied","Data":"efec69836718682e289958eb72b7b74d3b4346d4316b2267f9587ed8c202b7a6"} Oct 30 22:20:16.999381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:16.999292 4991 scope.go:122] "RemoveContainer" containerID="0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a" Oct 30 22:20:17.002397 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.002371 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" event={"ID":"7caf6588-e2a5-4f42-b401-2810e464570a","Type":"ContainerStarted","Data":"13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf"} Oct 30 22:20:17.002530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.002412 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" event={"ID":"7caf6588-e2a5-4f42-b401-2810e464570a","Type":"ContainerStarted","Data":"3eb7a3a18459b33e06b181a102126e208491f9b8b285bde340ecd1251fcc7f1f"} Oct 30 22:20:17.002530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.002492 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:17.003911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.003885 4991 generic.go:356] "Generic (PLEG): container finished" podID="b87c71bf-75f8-4a27-91cb-1547f7629af6" containerID="fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614" exitCode=0 Oct 30 22:20:17.004046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.003929 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" Oct 30 22:20:17.004046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.003950 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" event={"ID":"b87c71bf-75f8-4a27-91cb-1547f7629af6","Type":"ContainerDied","Data":"fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614"} Oct 30 22:20:17.004046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.003988 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng" event={"ID":"b87c71bf-75f8-4a27-91cb-1547f7629af6","Type":"ContainerDied","Data":"a3c56b2f16cfebd426778dadbc17071983f38d944a8bd6eaa234a90c18d53d0d"} Oct 30 22:20:17.011313 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.011297 4991 scope.go:122] "RemoveContainer" containerID="0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a" Oct 30 22:20:17.011599 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:17.011580 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a\": container with ID starting with 0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a not found: ID does not exist" containerID="0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a" Oct 30 22:20:17.011673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.011607 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a"} err="failed to get container status \"0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a\": rpc error: code = NotFound desc = could not find container \"0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a\": container with ID starting with 0951bd7694ce69ef4aef7f179be79f525d8687326db4638a6d66448165e21e2a not found: ID does not exist" Oct 30 22:20:17.011673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.011624 4991 scope.go:122] "RemoveContainer" containerID="fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614" Oct 30 22:20:17.027061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.026989 4991 scope.go:122] "RemoveContainer" containerID="fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614" Oct 30 22:20:17.027604 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:17.027404 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614\": container with ID starting with fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614 not found: ID does not exist" containerID="fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614" Oct 30 22:20:17.027604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.027437 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614"} err="failed to get container status \"fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614\": rpc error: code = NotFound desc = could not find container \"fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614\": container with ID starting with fb1bc0837cf83da557da3080b2ede3aa407af1f1ae2ffdc153775f6610c90614 not found: ID does not exist" Oct 30 22:20:17.157908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.157768 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng"] Oct 30 22:20:17.162500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.162470 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4576d9-5c76f5c8dd-twjng"] Oct 30 22:20:17.271385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.271305 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b"] Oct 30 22:20:17.274400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.274371 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-8658685c87-s8k7b"] Oct 30 22:20:17.379762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.379708 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" podStartSLOduration=2.379690581 podStartE2EDuration="2.379690581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:17.37907018 +0000 UTC m=+2037.678480062" watchObservedRunningTime="2026-10-30 22:20:17.379690581 +0000 UTC m=+2037.679100462" Oct 30 22:20:17.876653 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.876621 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" path="/var/lib/kubelet/pods/b22627eb-ff59-47c4-b21d-d49ff8e8c4fc/volumes" Oct 30 22:20:17.877123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:17.877108 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87c71bf-75f8-4a27-91cb-1547f7629af6" path="/var/lib/kubelet/pods/b87c71bf-75f8-4a27-91cb-1547f7629af6/volumes" Oct 30 22:20:17.911264 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:17.911233 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:20:19.475178 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:19.475148 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e8c5f9_bdc6_480d_afb5_af949b21b993.slice/crio-conmon-e7f5dce1c5d6ab668242b8bf1e4411cc3f1f3ed74e6b3301c15e0c7357e12d75.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:20:26.257645 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:26.257607 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t"] Oct 30 22:20:26.258526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:26.258489 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" containerID="cri-o://95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7" gracePeriod=30 Oct 30 22:20:28.016028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.015996 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:20:28.462932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.462850 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75"] Oct 30 22:20:28.463532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.463511 4991 cpu_manager.go:414] "containerMap: removing container" podUID="b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" containerName="payload-pre-processing" Oct 30 22:20:28.463532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.463533 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" containerName="payload-pre-processing" Oct 30 22:20:28.463713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.463578 4991 cpu_manager.go:414] "containerMap: removing container" podUID="b87c71bf-75f8-4a27-91cb-1547f7629af6" containerName="maas-api" Oct 30 22:20:28.463713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.463587 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87c71bf-75f8-4a27-91cb-1547f7629af6" containerName="maas-api" Oct 30 22:20:28.463823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.463737 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22627eb-ff59-47c4-b21d-d49ff8e8c4fc" containerName="payload-pre-processing" Oct 30 22:20:28.463823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.463771 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87c71bf-75f8-4a27-91cb-1547f7629af6" containerName="maas-api" Oct 30 22:20:28.476358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.476296 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.527711 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.527676 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3bd7fe-openshift-default-dockercfg-m6m6d\"" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.572056 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75"] Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.579683 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-certs\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.580673 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-credential-socket\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.580718 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1905bbf4-da83-45f1-addc-018a5e968d76-istiod-ca-cert\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.580766 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-envoy\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.580814 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-data\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.580848 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkzg\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-kube-api-access-gwkzg\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.580890 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-istio-token\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.580976 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1905bbf4-da83-45f1-addc-018a5e968d76-istio-podinfo\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.588480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.581023 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-socket\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682407 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-certs\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682490 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-credential-socket\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682519 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1905bbf4-da83-45f1-addc-018a5e968d76-istiod-ca-cert\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682548 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-envoy\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682587 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-data\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682616 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkzg\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-kube-api-access-gwkzg\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682654 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-istio-token\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682716 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1905bbf4-da83-45f1-addc-018a5e968d76-istio-podinfo\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.682913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682749 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-socket\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.683117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.682947 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-credential-socket\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.683117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.683075 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-data\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.683117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.683091 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-socket\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.683494 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.683473 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-certs\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.683919 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.683893 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1905bbf4-da83-45f1-addc-018a5e968d76-istiod-ca-cert\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.685615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.685591 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-envoy\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.685710 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.685615 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1905bbf4-da83-45f1-addc-018a5e968d76-istio-podinfo\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.741027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.740996 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-istio-token\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.741575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.741555 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkzg\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-kube-api-access-gwkzg\") pod \"e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:28.822004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:28.821961 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:29.276779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:29.276746 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75"] Oct 30 22:20:29.282962 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:20:29.282929 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1905bbf4_da83_45f1_addc_018a5e968d76.slice/crio-c9ab38f601c2511d8d14d2705cc9bf13e0efd8ab1f7c0ee71b7c2847c6c19fe8 WatchSource:0}: Error finding container c9ab38f601c2511d8d14d2705cc9bf13e0efd8ab1f7c0ee71b7c2847c6c19fe8: Status 404 returned error can't find the container with id c9ab38f601c2511d8d14d2705cc9bf13e0efd8ab1f7c0ee71b7c2847c6c19fe8 Oct 30 22:20:29.285297 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:29.285266 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:29.285392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:29.285344 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:29.285392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:29.285375 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:20:30.097458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:30.097416 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" event={"ID":"1905bbf4-da83-45f1-addc-018a5e968d76","Type":"ContainerStarted","Data":"16b7cb7a97ae1da1932d6e3d967bdc091f8143dfdef00a4a95c75bce6401e84f"} Oct 30 22:20:30.097631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:30.097481 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" event={"ID":"1905bbf4-da83-45f1-addc-018a5e968d76","Type":"ContainerStarted","Data":"c9ab38f601c2511d8d14d2705cc9bf13e0efd8ab1f7c0ee71b7c2847c6c19fe8"} Oct 30 22:20:30.313820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:30.313762 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" podStartSLOduration=2.313749508 podStartE2EDuration="2.313749508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:30.265004545 +0000 UTC m=+2050.564414425" watchObservedRunningTime="2026-10-30 22:20:30.313749508 +0000 UTC m=+2050.613159388" Oct 30 22:20:30.822835 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:30.822798 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:30.828065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:30.828040 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:31.104447 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.104372 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:31.105393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.105370 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:20:31.552214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.552184 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:20:31.619100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619073 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-credential-socket\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619144 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-podinfo\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619187 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-socket\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619230 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-certs\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619260 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-kube-api-access-sj99c\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-kube-api-access-sj99c\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619378 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-credential-socket" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:31.619457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619384 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0243d7ec-a527-4410-bb63-5eb4443d234d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0243d7ec-a527-4410-bb63-5eb4443d234d-istiod-ca-cert\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619491 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-socket" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:31.619623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619514 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-data\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619573 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-envoy\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619648 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-token\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-token\") pod \"0243d7ec-a527-4410-bb63-5eb4443d234d\" (UID: \"0243d7ec-a527-4410-bb63-5eb4443d234d\") " Oct 30 22:20:31.619786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619655 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-certs" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:31.619957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619823 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-data" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:31.619957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.619931 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0243d7ec-a527-4410-bb63-5eb4443d234d-istiod-ca-cert" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:31.621727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.621702 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-podinfo" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:31.622030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622003 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-token" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:31.622113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622046 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-kube-api-access-sj99c" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "kube-api-access-sj99c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:31.622161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622113 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-envoy" pod "0243d7ec-a527-4410-bb63-5eb4443d234d" (UID: "0243d7ec-a527-4410-bb63-5eb4443d234d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:31.622785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622764 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622793 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622807 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622821 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622837 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0243d7ec-a527-4410-bb63-5eb4443d234d-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622853 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622866 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0243d7ec-a527-4410-bb63-5eb4443d234d-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622879 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj99c\" (UniqueName: \"kubernetes.io/projected/0243d7ec-a527-4410-bb63-5eb4443d234d-kube-api-access-sj99c\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.622978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:31.622894 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0243d7ec-a527-4410-bb63-5eb4443d234d-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:32.111776 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.111742 4991 generic.go:356] "Generic (PLEG): container finished" podID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerID="95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7" exitCode=0 Oct 30 22:20:32.111957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.111797 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" Oct 30 22:20:32.111957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.111818 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" event={"ID":"0243d7ec-a527-4410-bb63-5eb4443d234d","Type":"ContainerDied","Data":"95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7"} Oct 30 22:20:32.111957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.111872 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t" event={"ID":"0243d7ec-a527-4410-bb63-5eb4443d234d","Type":"ContainerDied","Data":"a5dc9946b31f36026a71161b198e37da50a31383ae3c590f6db9669f664260bd"} Oct 30 22:20:32.111957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.111893 4991 scope.go:122] "RemoveContainer" containerID="95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7" Oct 30 22:20:32.122298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.122280 4991 scope.go:122] "RemoveContainer" containerID="12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d" Oct 30 22:20:32.131226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.131209 4991 scope.go:122] "RemoveContainer" containerID="95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7" Oct 30 22:20:32.131513 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:32.131493 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7\": container with ID starting with 95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7 not found: ID does not exist" containerID="95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7" Oct 30 22:20:32.131611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.131521 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7"} err="failed to get container status \"95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7\": rpc error: code = NotFound desc = could not find container \"95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7\": container with ID starting with 95eb26118a447c9d4d84f120c7ee16e5ebdfac025fd1599e79567c9ebc9893d7 not found: ID does not exist" Oct 30 22:20:32.131611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.131537 4991 scope.go:122] "RemoveContainer" containerID="12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d" Oct 30 22:20:32.131798 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:32.131780 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d\": container with ID starting with 12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d not found: ID does not exist" containerID="12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d" Oct 30 22:20:32.131848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.131804 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d"} err="failed to get container status \"12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d\": rpc error: code = NotFound desc = could not find container \"12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d\": container with ID starting with 12719322e105b8608c18ec9842fda1f143f982e0313869ec4195bdffabd92c0d not found: ID does not exist" Oct 30 22:20:32.269488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.269458 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t"] Oct 30 22:20:32.315670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:32.315641 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4576d9-openshift-default-dbcdcf94c-5b94t"] Oct 30 22:20:33.875792 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:33.875759 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" path="/var/lib/kubelet/pods/0243d7ec-a527-4410-bb63-5eb4443d234d/volumes" Oct 30 22:20:41.591740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:41.591710 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps"] Oct 30 22:20:41.592166 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:41.591905 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" containerID="cri-o://11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1" gracePeriod=30 Oct 30 22:20:46.862040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.861971 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:20:46.926377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926351 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1911928-972b-4ed5-8bd9-2acd10bb58be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1911928-972b-4ed5-8bd9-2acd10bb58be-istiod-ca-cert\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926392 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-token\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-token\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926415 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-envoy\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926447 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-credential-socket\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926545 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-certs\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926632 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-data\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926670 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-kube-api-access-6v6kk\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-kube-api-access-6v6kk\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926737 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-podinfo\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.926876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926800 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-credential-socket" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.926876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926825 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-socket\") pod \"f1911928-972b-4ed5-8bd9-2acd10bb58be\" (UID: \"f1911928-972b-4ed5-8bd9-2acd10bb58be\") " Oct 30 22:20:46.927154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.926821 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1911928-972b-4ed5-8bd9-2acd10bb58be-istiod-ca-cert" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:46.927154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.927123 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-socket" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.927389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.927358 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.927512 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.927390 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.927512 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.927407 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1911928-972b-4ed5-8bd9-2acd10bb58be-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:46.927645 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.927587 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-data" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.927645 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.927605 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-certs" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.928819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.928789 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-envoy" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:46.929173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.929147 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-token" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:46.929173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.929159 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-kube-api-access-6v6kk" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "kube-api-access-6v6kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:46.929280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:46.929236 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-podinfo" pod "f1911928-972b-4ed5-8bd9-2acd10bb58be" (UID: "f1911928-972b-4ed5-8bd9-2acd10bb58be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:47.028459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.028426 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:47.028459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.028457 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:47.028646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.028472 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:47.028646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.028487 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:47.028646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.028502 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6kk\" (UniqueName: \"kubernetes.io/projected/f1911928-972b-4ed5-8bd9-2acd10bb58be-kube-api-access-6v6kk\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:47.028646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.028517 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1911928-972b-4ed5-8bd9-2acd10bb58be-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:47.224428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.224392 4991 generic.go:356] "Generic (PLEG): container finished" podID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerID="11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1" exitCode=0 Oct 30 22:20:47.224596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.224466 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" Oct 30 22:20:47.224596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.224476 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" event={"ID":"f1911928-972b-4ed5-8bd9-2acd10bb58be","Type":"ContainerDied","Data":"11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1"} Oct 30 22:20:47.224596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.224543 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps" event={"ID":"f1911928-972b-4ed5-8bd9-2acd10bb58be","Type":"ContainerDied","Data":"42bba93c1dc7951b8b71ca11f63e5976daab1fd9182e11b2339865b120045a0d"} Oct 30 22:20:47.224596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.224574 4991 scope.go:122] "RemoveContainer" containerID="11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1" Oct 30 22:20:47.237097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.237076 4991 scope.go:122] "RemoveContainer" containerID="04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3" Oct 30 22:20:47.246545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.246525 4991 scope.go:122] "RemoveContainer" containerID="11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1" Oct 30 22:20:47.246839 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:47.246805 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1\": container with ID starting with 11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1 not found: ID does not exist" containerID="11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1" Oct 30 22:20:47.246907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.246847 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1"} err="failed to get container status \"11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1\": rpc error: code = NotFound desc = could not find container \"11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1\": container with ID starting with 11dc9edfd36c52a54a91bf8be166534b4cb69ae5f0327568df8af03db7441eb1 not found: ID does not exist" Oct 30 22:20:47.246907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.246865 4991 scope.go:122] "RemoveContainer" containerID="04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3" Oct 30 22:20:47.247104 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:47.247086 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3\": container with ID starting with 04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3 not found: ID does not exist" containerID="04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3" Oct 30 22:20:47.247162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.247112 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3"} err="failed to get container status \"04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3\": rpc error: code = NotFound desc = could not find container \"04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3\": container with ID starting with 04d999a98cf26e7c285f3c686b9f58772aac817442a4fce08b36a20fe938e1c3 not found: ID does not exist" Oct 30 22:20:47.383866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.383831 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps"] Oct 30 22:20:47.388208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.388186 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b4b54-openshift-default-844b4bc955-4d6ps"] Oct 30 22:20:47.876641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:47.876604 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" path="/var/lib/kubelet/pods/f1911928-972b-4ed5-8bd9-2acd10bb58be/volumes" Oct 30 22:20:47.910306 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:20:47.910272 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:20:52.966432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966393 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz"] Oct 30 22:20:52.966946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966910 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" Oct 30 22:20:52.966946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966922 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" Oct 30 22:20:52.966946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966944 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966950 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966957 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966962 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966972 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.966978 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.967046 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.967054 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" Oct 30 22:20:52.967068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.967060 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0243d7ec-a527-4410-bb63-5eb4443d234d" containerName="istio-proxy" Oct 30 22:20:52.972294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:52.972273 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.068436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.068402 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0e90fe\"" Oct 30 22:20:53.072341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.072305 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz"] Oct 30 22:20:53.097490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.097465 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/fd453ded-f246-4c99-aca7-a2a4d3434eb3-kube-api-access-xstl7\") pod \"payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.097681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.097661 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd453ded-f246-4c99-aca7-a2a4d3434eb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.198604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.198548 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd453ded-f246-4c99-aca7-a2a4d3434eb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.198811 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.198660 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/fd453ded-f246-4c99-aca7-a2a4d3434eb3-kube-api-access-xstl7\") pod \"payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.200011 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.199981 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd453ded-f246-4c99-aca7-a2a4d3434eb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.257281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.257246 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/fd453ded-f246-4c99-aca7-a2a4d3434eb3-kube-api-access-xstl7\") pod \"payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.286512 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.286487 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:53.418998 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.418964 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv"] Oct 30 22:20:53.419750 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.419727 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1911928-972b-4ed5-8bd9-2acd10bb58be" containerName="istio-proxy" Oct 30 22:20:53.424163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.424142 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:53.475386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.475168 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0e90fe-dockercfg-5kjh5\"" Oct 30 22:20:53.479391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.479369 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv"] Oct 30 22:20:53.501753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.501728 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c88f73-6294-4180-a08a-859f1090acf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:53.501968 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.501939 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9r2g\" (UniqueName: \"kubernetes.io/projected/87c88f73-6294-4180-a08a-859f1090acf8-kube-api-access-b9r2g\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:53.602771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.602707 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9r2g\" (UniqueName: \"kubernetes.io/projected/87c88f73-6294-4180-a08a-859f1090acf8-kube-api-access-b9r2g\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:53.602884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.602774 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c88f73-6294-4180-a08a-859f1090acf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:53.603512 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.603492 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c88f73-6294-4180-a08a-859f1090acf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:53.661046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.661020 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9r2g\" (UniqueName: \"kubernetes.io/projected/87c88f73-6294-4180-a08a-859f1090acf8-kube-api-access-b9r2g\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:53.681449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.681426 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz"] Oct 30 22:20:53.687558 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:20:53.687531 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd453ded_f246_4c99_aca7_a2a4d3434eb3.slice/crio-6a9be8ab8b92fda87eb0499ab2f9902ea281339471bbf4759c558089bf5e23af WatchSource:0}: Error finding container 6a9be8ab8b92fda87eb0499ab2f9902ea281339471bbf4759c558089bf5e23af: Status 404 returned error can't find the container with id 6a9be8ab8b92fda87eb0499ab2f9902ea281339471bbf4759c558089bf5e23af Oct 30 22:20:53.738612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:53.738588 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:54.119601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.119567 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv"] Oct 30 22:20:54.126262 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:20:54.126232 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c88f73_6294_4180_a08a_859f1090acf8.slice/crio-57d05a578f61b2028a8ebfa99c8ea30634423efbda1aef9512dbd53153518b44 WatchSource:0}: Error finding container 57d05a578f61b2028a8ebfa99c8ea30634423efbda1aef9512dbd53153518b44: Status 404 returned error can't find the container with id 57d05a578f61b2028a8ebfa99c8ea30634423efbda1aef9512dbd53153518b44 Oct 30 22:20:54.315262 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.315221 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" event={"ID":"87c88f73-6294-4180-a08a-859f1090acf8","Type":"ContainerStarted","Data":"082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443"} Oct 30 22:20:54.315476 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.315315 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" event={"ID":"87c88f73-6294-4180-a08a-859f1090acf8","Type":"ContainerStarted","Data":"57d05a578f61b2028a8ebfa99c8ea30634423efbda1aef9512dbd53153518b44"} Oct 30 22:20:54.315476 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.315370 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:20:54.317211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.317173 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" event={"ID":"fd453ded-f246-4c99-aca7-a2a4d3434eb3","Type":"ContainerStarted","Data":"4de61207d7ab0a78fbdd338aeb9f469fb23d4c067fb423946cd48c8379d2de13"} Oct 30 22:20:54.317370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.317231 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" event={"ID":"fd453ded-f246-4c99-aca7-a2a4d3434eb3","Type":"ContainerStarted","Data":"6a9be8ab8b92fda87eb0499ab2f9902ea281339471bbf4759c558089bf5e23af"} Oct 30 22:20:54.317370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.317313 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:20:54.425204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.425095 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" podStartSLOduration=1.4250789560000001 podStartE2EDuration="1.425078956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:54.424732005 +0000 UTC m=+2074.724141905" watchObservedRunningTime="2026-10-30 22:20:54.425078956 +0000 UTC m=+2074.724488836" Oct 30 22:20:54.547337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:20:54.546633 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" podStartSLOduration=2.5466087059999998 podStartE2EDuration="2.546608706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:54.535246578 +0000 UTC m=+2074.834656459" watchObservedRunningTime="2026-10-30 22:20:54.546608706 +0000 UTC m=+2074.846018621" Oct 30 22:21:05.326972 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.326941 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:21:05.327449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.327008 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:21:05.492259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.492229 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v"] Oct 30 22:21:05.497121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.497096 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.534651 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.534625 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.534789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.534664 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.534789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.534759 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.534882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.534808 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpzc\" (UniqueName: \"kubernetes.io/projected/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-kube-api-access-8xpzc\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.548452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.548421 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6a3357-serving-cert\"" Oct 30 22:21:05.555794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.555768 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v"] Oct 30 22:21:05.635953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.635864 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.636379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.636352 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.636505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.636408 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.636630 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:21:05.636614 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-serving-cert: secret "maas-api-e2e-ipp-b-6a3357-serving-cert" not found Oct 30 22:21:05.637456 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:21:05.637427 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls podName:9eb25b59-8b56-4d2d-bb55-6c9854e1624d nodeName:}" failed. No retries permitted until 2026-10-30 22:21:06.137402413 +0000 UTC m=+2086.436812286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls") pod "maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" (UID: "9eb25b59-8b56-4d2d-bb55-6c9854e1624d") : secret "maas-api-e2e-ipp-b-6a3357-serving-cert" not found Oct 30 22:21:05.637580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.637549 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpzc\" (UniqueName: \"kubernetes.io/projected/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-kube-api-access-8xpzc\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.638224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.638201 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.638813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.638795 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:05.696350 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:05.696290 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpzc\" (UniqueName: \"kubernetes.io/projected/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-kube-api-access-8xpzc\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:06.146807 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:06.146776 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:06.149203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:06.149177 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:06.412416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:06.412337 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:06.808521 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:06.808492 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v"] Oct 30 22:21:06.813556 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:21:06.813529 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb25b59_8b56_4d2d_bb55_6c9854e1624d.slice/crio-c7c36ba5d2fdd856a8d34e9687eb72f79f584c33af86b526c105ae8ab5a23514 WatchSource:0}: Error finding container c7c36ba5d2fdd856a8d34e9687eb72f79f584c33af86b526c105ae8ab5a23514: Status 404 returned error can't find the container with id c7c36ba5d2fdd856a8d34e9687eb72f79f584c33af86b526c105ae8ab5a23514 Oct 30 22:21:06.814896 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:06.814879 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:21:07.412174 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:07.412138 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" event={"ID":"9eb25b59-8b56-4d2d-bb55-6c9854e1624d","Type":"ContainerStarted","Data":"c7c36ba5d2fdd856a8d34e9687eb72f79f584c33af86b526c105ae8ab5a23514"} Oct 30 22:21:08.432559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:08.432519 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" event={"ID":"9eb25b59-8b56-4d2d-bb55-6c9854e1624d","Type":"ContainerStarted","Data":"484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe"} Oct 30 22:21:08.433531 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:08.433246 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:08.553654 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:08.553587 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" podStartSLOduration=2.685846576 podStartE2EDuration="3.553565878s" totalImagesPullingTime="867.719302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:08.551187346 +0000 UTC m=+2088.850597246" watchObservedRunningTime="2026-10-30 22:21:08.553565878 +0000 UTC m=+2088.852975759" Oct 30 22:21:15.451617 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:15.451587 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:21:17.911424 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:21:17.911386 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:21:20.235281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:20.235249 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:21:20.258367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:20.258343 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:21:24.999219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:24.999189 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j"] Oct 30 22:21:25.005969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.005944 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.062635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.062606 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j"] Oct 30 22:21:25.103734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.103707 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3bd7fe-serving-cert\"" Oct 30 22:21:25.104427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.104403 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbj78\" (UniqueName: \"kubernetes.io/projected/de363edf-a0ea-4b46-b80c-53157e82dbf5-kube-api-access-pbj78\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.104542 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.104437 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.104542 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.104516 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de363edf-a0ea-4b46-b80c-53157e82dbf5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.104624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.104563 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-metrics-certs\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.206183 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.206150 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbj78\" (UniqueName: \"kubernetes.io/projected/de363edf-a0ea-4b46-b80c-53157e82dbf5-kube-api-access-pbj78\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.206376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.206189 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.206376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.206235 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de363edf-a0ea-4b46-b80c-53157e82dbf5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.206376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.206281 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-metrics-certs\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.206566 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:21:25.206545 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-serving-cert: secret "maas-api-e2e-praxis-a-3bd7fe-serving-cert" not found Oct 30 22:21:25.206672 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:21:25.206616 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls podName:de363edf-a0ea-4b46-b80c-53157e82dbf5 nodeName:}" failed. No retries permitted until 2026-10-30 22:21:25.706594262 +0000 UTC m=+2106.006004141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls") pod "maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" (UID: "de363edf-a0ea-4b46-b80c-53157e82dbf5") : secret "maas-api-e2e-praxis-a-3bd7fe-serving-cert" not found Oct 30 22:21:25.207201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.207176 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de363edf-a0ea-4b46-b80c-53157e82dbf5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.208724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.208704 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-metrics-certs\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.275208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.275064 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbj78\" (UniqueName: \"kubernetes.io/projected/de363edf-a0ea-4b46-b80c-53157e82dbf5-kube-api-access-pbj78\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.713013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.712930 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.715268 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.715245 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:25.938343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:25.923527 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:26.357493 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:26.357445 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j"] Oct 30 22:21:26.364635 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:21:26.364605 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde363edf_a0ea_4b46_b80c_53157e82dbf5.slice/crio-3cf4f4063743c67a3b7dacdbaec6668332fab5153c0f110b4090ce1764804c86 WatchSource:0}: Error finding container 3cf4f4063743c67a3b7dacdbaec6668332fab5153c0f110b4090ce1764804c86: Status 404 returned error can't find the container with id 3cf4f4063743c67a3b7dacdbaec6668332fab5153c0f110b4090ce1764804c86 Oct 30 22:21:26.592315 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:26.592277 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" event={"ID":"de363edf-a0ea-4b46-b80c-53157e82dbf5","Type":"ContainerStarted","Data":"3cf4f4063743c67a3b7dacdbaec6668332fab5153c0f110b4090ce1764804c86"} Oct 30 22:21:27.599616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:27.599536 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" event={"ID":"de363edf-a0ea-4b46-b80c-53157e82dbf5","Type":"ContainerStarted","Data":"645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c"} Oct 30 22:21:27.599616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:27.599610 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:27.719912 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:27.719849 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" podStartSLOduration=2.747380498 podStartE2EDuration="3.719833891s" totalImagesPullingTime="972.453393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:27.715600331 +0000 UTC m=+2108.015010210" watchObservedRunningTime="2026-10-30 22:21:27.719833891 +0000 UTC m=+2108.019243770" Oct 30 22:21:30.620350 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:30.620285 4991 generic.go:356] "Generic (PLEG): container finished" podID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerID="426319d52a16627dfa1ba4fa21727cb49d8f386e3dac7537b6c50bd756761994" exitCode=0 Oct 30 22:21:30.620779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:30.620354 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" event={"ID":"0c0e72d9-7a32-4f56-b382-c998e1bf046f","Type":"ContainerDied","Data":"426319d52a16627dfa1ba4fa21727cb49d8f386e3dac7537b6c50bd756761994"} Oct 30 22:21:30.620779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:30.620728 4991 scope.go:122] "RemoveContainer" containerID="426319d52a16627dfa1ba4fa21727cb49d8f386e3dac7537b6c50bd756761994" Oct 30 22:21:30.622165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:30.622133 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:30.622289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:30.622215 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:30.622289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:30.622262 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:31.651701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:31.651667 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" event={"ID":"0c0e72d9-7a32-4f56-b382-c998e1bf046f","Type":"ContainerStarted","Data":"1896233dfe3fbd615365702b549826b6f91a9d2b726259f779166851dd657784"} Oct 30 22:21:32.251633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:32.251600 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:21:32.251903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:32.251880 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:21:32.251949 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:32.251927 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:21:33.252067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:33.252032 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:21:33.252524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:33.252095 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:21:33.611235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:33.611157 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:21:34.251847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:34.251814 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:21:34.252038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:34.251879 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:21:35.252028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:35.251991 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:21:35.252433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:35.252064 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:21:36.252344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:36.251565 4991 patch_prober.go:29] interesting pod/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:21:36.252344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:36.251646 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 22:21:37.256637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:37.256602 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:21:37.257050 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:37.256973 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:21:37.257753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:37.257737 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:21:47.913901 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:21:47.913867 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:21:57.849897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:57.849860 4991 generic.go:356] "Generic (PLEG): container finished" podID="1905bbf4-da83-45f1-addc-018a5e968d76" containerID="16b7cb7a97ae1da1932d6e3d967bdc091f8143dfdef00a4a95c75bce6401e84f" exitCode=0 Oct 30 22:21:57.850458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:57.849916 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" event={"ID":"1905bbf4-da83-45f1-addc-018a5e968d76","Type":"ContainerDied","Data":"16b7cb7a97ae1da1932d6e3d967bdc091f8143dfdef00a4a95c75bce6401e84f"} Oct 30 22:21:57.850458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:57.850304 4991 scope.go:122] "RemoveContainer" containerID="16b7cb7a97ae1da1932d6e3d967bdc091f8143dfdef00a4a95c75bce6401e84f" Oct 30 22:21:57.851789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:57.851760 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:57.851875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:57.851830 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:57.851875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:57.851864 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:21:58.624226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.624182 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f"] Oct 30 22:21:58.624427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.624406 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" podUID="45fdf4a4-9485-4182-adb6-0352a25f3a1e" containerName="maas-api" containerID="cri-o://432588e8034b76c7a75c397c473e119288e82eed6583939c1fe4edc77c3d5ccc" gracePeriod=30 Oct 30 22:21:58.868906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.868358 4991 generic.go:356] "Generic (PLEG): container finished" podID="45fdf4a4-9485-4182-adb6-0352a25f3a1e" containerID="432588e8034b76c7a75c397c473e119288e82eed6583939c1fe4edc77c3d5ccc" exitCode=0 Oct 30 22:21:58.868906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.868586 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" event={"ID":"45fdf4a4-9485-4182-adb6-0352a25f3a1e","Type":"ContainerDied","Data":"432588e8034b76c7a75c397c473e119288e82eed6583939c1fe4edc77c3d5ccc"} Oct 30 22:21:58.873305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.872649 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" event={"ID":"1905bbf4-da83-45f1-addc-018a5e968d76","Type":"ContainerStarted","Data":"afdf02e98516598e492dcfc578ec9c86bfe6247fa2dfd378bff96856f685a40a"} Oct 30 22:21:58.915849 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.915830 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:21:58.974863 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.974838 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fdf4a4-9485-4182-adb6-0352a25f3a1e-kube-api-access-f56m9\" (UniqueName: \"kubernetes.io/projected/45fdf4a4-9485-4182-adb6-0352a25f3a1e-kube-api-access-f56m9\") pod \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " Oct 30 22:21:58.974990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.974904 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45fdf4a4-9485-4182-adb6-0352a25f3a1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45fdf4a4-9485-4182-adb6-0352a25f3a1e-openshift-service-ca\") pod \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " Oct 30 22:21:58.975036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.975002 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-metrics-certs\") pod \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " Oct 30 22:21:58.975073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.975064 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls\") pod \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\" (UID: \"45fdf4a4-9485-4182-adb6-0352a25f3a1e\") " Oct 30 22:21:58.975263 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.975229 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fdf4a4-9485-4182-adb6-0352a25f3a1e-openshift-service-ca" pod "45fdf4a4-9485-4182-adb6-0352a25f3a1e" (UID: "45fdf4a4-9485-4182-adb6-0352a25f3a1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:58.975840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.975815 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45fdf4a4-9485-4182-adb6-0352a25f3a1e-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:58.977388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.977293 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fdf4a4-9485-4182-adb6-0352a25f3a1e-kube-api-access-f56m9" pod "45fdf4a4-9485-4182-adb6-0352a25f3a1e" (UID: "45fdf4a4-9485-4182-adb6-0352a25f3a1e"). InnerVolumeSpecName "kube-api-access-f56m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:58.977692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.977675 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls" pod "45fdf4a4-9485-4182-adb6-0352a25f3a1e" (UID: "45fdf4a4-9485-4182-adb6-0352a25f3a1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:58.977771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:58.977739 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-metrics-certs" pod "45fdf4a4-9485-4182-adb6-0352a25f3a1e" (UID: "45fdf4a4-9485-4182-adb6-0352a25f3a1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:59.059830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.059786 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj"] Oct 30 22:21:59.060107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.060079 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" podUID="7caf6588-e2a5-4f42-b401-2810e464570a" containerName="payload-processing" containerID="cri-o://13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf" gracePeriod=30 Oct 30 22:21:59.079082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.079052 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:59.079242 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.079226 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f56m9\" (UniqueName: \"kubernetes.io/projected/45fdf4a4-9485-4182-adb6-0352a25f3a1e-kube-api-access-f56m9\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:59.079363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.079349 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fdf4a4-9485-4182-adb6-0352a25f3a1e-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:59.385615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.385585 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:21:59.483347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.483304 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7caf6588-e2a5-4f42-b401-2810e464570a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7caf6588-e2a5-4f42-b401-2810e464570a-plugins-config-volume\") pod \"7caf6588-e2a5-4f42-b401-2810e464570a\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " Oct 30 22:21:59.483504 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.483353 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7caf6588-e2a5-4f42-b401-2810e464570a-kube-api-access-wr2wx\" (UniqueName: \"kubernetes.io/projected/7caf6588-e2a5-4f42-b401-2810e464570a-kube-api-access-wr2wx\") pod \"7caf6588-e2a5-4f42-b401-2810e464570a\" (UID: \"7caf6588-e2a5-4f42-b401-2810e464570a\") " Oct 30 22:21:59.483678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.483655 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7caf6588-e2a5-4f42-b401-2810e464570a-plugins-config-volume" pod "7caf6588-e2a5-4f42-b401-2810e464570a" (UID: "7caf6588-e2a5-4f42-b401-2810e464570a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:59.483826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.483812 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7caf6588-e2a5-4f42-b401-2810e464570a-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:59.485426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.485405 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caf6588-e2a5-4f42-b401-2810e464570a-kube-api-access-wr2wx" pod "7caf6588-e2a5-4f42-b401-2810e464570a" (UID: "7caf6588-e2a5-4f42-b401-2810e464570a"). InnerVolumeSpecName "kube-api-access-wr2wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:59.595587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.595553 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2wx\" (UniqueName: \"kubernetes.io/projected/7caf6588-e2a5-4f42-b401-2810e464570a-kube-api-access-wr2wx\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:59.839398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.823605 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:21:59.850375 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.846093 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:21:59.850375 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.846188 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:21:59.882359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.882301 4991 generic.go:356] "Generic (PLEG): container finished" podID="7caf6588-e2a5-4f42-b401-2810e464570a" containerID="13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf" exitCode=0 Oct 30 22:21:59.882829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.882369 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" event={"ID":"7caf6588-e2a5-4f42-b401-2810e464570a","Type":"ContainerDied","Data":"13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf"} Oct 30 22:21:59.882829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.882461 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" event={"ID":"7caf6588-e2a5-4f42-b401-2810e464570a","Type":"ContainerDied","Data":"3eb7a3a18459b33e06b181a102126e208491f9b8b285bde340ecd1251fcc7f1f"} Oct 30 22:21:59.882829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.882492 4991 scope.go:122] "RemoveContainer" containerID="13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf" Oct 30 22:21:59.882829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.882709 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj" Oct 30 22:21:59.885699 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.885680 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" Oct 30 22:21:59.885779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.885715 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f" event={"ID":"45fdf4a4-9485-4182-adb6-0352a25f3a1e","Type":"ContainerDied","Data":"6b1dccacf6a4f250ae44a9302301fc52744f22d7f472b566ad01b7d8b901329b"} Oct 30 22:21:59.898855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.898837 4991 scope.go:122] "RemoveContainer" containerID="13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf" Oct 30 22:21:59.899127 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:21:59.899109 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf\": container with ID starting with 13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf not found: ID does not exist" containerID="13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf" Oct 30 22:21:59.899196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.899135 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf"} err="failed to get container status \"13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf\": rpc error: code = NotFound desc = could not find container \"13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf\": container with ID starting with 13ff7c9cbb8895d65cb6cfd106162ba6e8fd3756f3679888f1b1e47814e4f4bf not found: ID does not exist" Oct 30 22:21:59.899196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:21:59.899151 4991 scope.go:122] "RemoveContainer" containerID="432588e8034b76c7a75c397c473e119288e82eed6583939c1fe4edc77c3d5ccc" Oct 30 22:22:00.050432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:00.050380 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f"] Oct 30 22:22:00.054519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:00.054484 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8daffd-57b86d7d59-n9t7f"] Oct 30 22:22:00.156028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:00.155942 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj"] Oct 30 22:22:00.160698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:00.160670 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-8b48d74d4-wvtwj"] Oct 30 22:22:00.825345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:00.823617 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:22:00.825345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:00.823691 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:22:01.104729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:01.104521 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:22:01.822482 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:01.822444 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:22:01.822775 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:01.822745 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:22:01.875990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:01.875949 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fdf4a4-9485-4182-adb6-0352a25f3a1e" path="/var/lib/kubelet/pods/45fdf4a4-9485-4182-adb6-0352a25f3a1e/volumes" Oct 30 22:22:01.876390 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:01.876377 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7caf6588-e2a5-4f42-b401-2810e464570a" path="/var/lib/kubelet/pods/7caf6588-e2a5-4f42-b401-2810e464570a/volumes" Oct 30 22:22:02.843460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:02.843420 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:22:02.844062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:02.844028 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:22:03.824340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:03.823682 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:22:03.824340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:03.823749 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:22:04.828290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:04.828252 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:22:04.829534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:04.829510 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:22:11.116622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.116582 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j"] Oct 30 22:22:11.117540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.117514 4991 cpu_manager.go:414] "containerMap: removing container" podUID="45fdf4a4-9485-4182-adb6-0352a25f3a1e" containerName="maas-api" Oct 30 22:22:11.117540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.117539 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fdf4a4-9485-4182-adb6-0352a25f3a1e" containerName="maas-api" Oct 30 22:22:11.117741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.117576 4991 cpu_manager.go:414] "containerMap: removing container" podUID="7caf6588-e2a5-4f42-b401-2810e464570a" containerName="payload-processing" Oct 30 22:22:11.117741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.117585 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7caf6588-e2a5-4f42-b401-2810e464570a" containerName="payload-processing" Oct 30 22:22:11.117741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.117709 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7caf6588-e2a5-4f42-b401-2810e464570a" containerName="payload-processing" Oct 30 22:22:11.117741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.117729 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fdf4a4-9485-4182-adb6-0352a25f3a1e" containerName="maas-api" Oct 30 22:22:11.122146 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.122123 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.174591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.174560 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j"] Oct 30 22:22:11.178064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.178036 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1846b75c-df63-43f5-bbe3-6ce68ac286d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.178380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.178358 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7c2x\" (UniqueName: \"kubernetes.io/projected/1846b75c-df63-43f5-bbe3-6ce68ac286d6-kube-api-access-k7c2x\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.222377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.215590 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7350bc\"" Oct 30 22:22:11.222377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.215613 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7350bc-dockercfg-jwxm5\"" Oct 30 22:22:11.281031 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.280991 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1846b75c-df63-43f5-bbe3-6ce68ac286d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.281357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.281290 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7c2x\" (UniqueName: \"kubernetes.io/projected/1846b75c-df63-43f5-bbe3-6ce68ac286d6-kube-api-access-k7c2x\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.283526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.283446 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1846b75c-df63-43f5-bbe3-6ce68ac286d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.340473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.340442 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7c2x\" (UniqueName: \"kubernetes.io/projected/1846b75c-df63-43f5-bbe3-6ce68ac286d6-kube-api-access-k7c2x\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.469713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.469645 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.852093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.852060 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j"] Oct 30 22:22:11.859242 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:22:11.859212 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1846b75c_df63_43f5_bbe3_6ce68ac286d6.slice/crio-b4ae75705800aa5236628afad2044b340d93254aa4802cfe93e1b41bcf31bc60 WatchSource:0}: Error finding container b4ae75705800aa5236628afad2044b340d93254aa4802cfe93e1b41bcf31bc60: Status 404 returned error can't find the container with id b4ae75705800aa5236628afad2044b340d93254aa4802cfe93e1b41bcf31bc60 Oct 30 22:22:11.876165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.876144 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq"] Oct 30 22:22:11.879801 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.879782 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:11.957695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.957670 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq"] Oct 30 22:22:11.976467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.976439 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" event={"ID":"1846b75c-df63-43f5-bbe3-6ce68ac286d6","Type":"ContainerStarted","Data":"6f6ce78df12ff10c2253ff65c2f5231e9044ca87243dfb834d7695922751280e"} Oct 30 22:22:11.976596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.976489 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" event={"ID":"1846b75c-df63-43f5-bbe3-6ce68ac286d6","Type":"ContainerStarted","Data":"b4ae75705800aa5236628afad2044b340d93254aa4802cfe93e1b41bcf31bc60"} Oct 30 22:22:11.976596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.976519 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:11.990867 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.990839 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5tz\" (UniqueName: \"kubernetes.io/projected/3209dbf8-21b4-4e34-ac01-ced869752247-kube-api-access-8k5tz\") pod \"payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:11.991090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:11.991069 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3209dbf8-21b4-4e34-ac01-ced869752247-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:12.092785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.092756 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5tz\" (UniqueName: \"kubernetes.io/projected/3209dbf8-21b4-4e34-ac01-ced869752247-kube-api-access-8k5tz\") pod \"payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:12.093257 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.092846 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3209dbf8-21b4-4e34-ac01-ced869752247-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:12.093588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.093568 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3209dbf8-21b4-4e34-ac01-ced869752247-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:12.099679 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.099630 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" podStartSLOduration=1.099614543 podStartE2EDuration="1.099614543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:12.098064683 +0000 UTC m=+2152.397474589" watchObservedRunningTime="2026-10-30 22:22:12.099614543 +0000 UTC m=+2152.399024424" Oct 30 22:22:12.154535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.154463 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5tz\" (UniqueName: \"kubernetes.io/projected/3209dbf8-21b4-4e34-ac01-ced869752247-kube-api-access-8k5tz\") pod \"payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:12.213799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.213770 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:12.664796 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.664764 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq"] Oct 30 22:22:12.670585 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:22:12.670557 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3209dbf8_21b4_4e34_ac01_ced869752247.slice/crio-feef1c0e5c58ceae44499d7a21660215cbf8b191138c3efbc30ba5f7dd420c64 WatchSource:0}: Error finding container feef1c0e5c58ceae44499d7a21660215cbf8b191138c3efbc30ba5f7dd420c64: Status 404 returned error can't find the container with id feef1c0e5c58ceae44499d7a21660215cbf8b191138c3efbc30ba5f7dd420c64 Oct 30 22:22:12.984522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.984488 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" event={"ID":"3209dbf8-21b4-4e34-ac01-ced869752247","Type":"ContainerStarted","Data":"6e21152ab93c6177a329a400ed302714e2d755531b1219b70121f48140c8a855"} Oct 30 22:22:12.984710 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.984574 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" event={"ID":"3209dbf8-21b4-4e34-ac01-ced869752247","Type":"ContainerStarted","Data":"feef1c0e5c58ceae44499d7a21660215cbf8b191138c3efbc30ba5f7dd420c64"} Oct 30 22:22:12.984710 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:12.984604 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:13.099606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:13.099547 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" podStartSLOduration=2.099532709 podStartE2EDuration="2.099532709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:13.096792714 +0000 UTC m=+2153.396202613" watchObservedRunningTime="2026-10-30 22:22:13.099532709 +0000 UTC m=+2153.398942588" Oct 30 22:22:17.910289 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:22:17.910254 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:22:22.986657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:22.986627 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:22:23.995027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:23.994999 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:22:29.891970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:29.891934 4991 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod7caf6588-e2a5-4f42-b401-2810e464570a"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod7caf6588-e2a5-4f42-b401-2810e464570a] : Timed out while waiting for systemd to remove kubepods-burstable-pod7caf6588_e2a5_4f42_b401_2810e464570a.slice" Oct 30 22:22:29.891970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:22:29.891966 4991 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="7caf6588-e2a5-4f42-b401-2810e464570a" err="failed to delete cgroup paths for [kubepods burstable pod7caf6588-e2a5-4f42-b401-2810e464570a] : unable to destroy cgroup paths for cgroup [kubepods burstable pod7caf6588-e2a5-4f42-b401-2810e464570a] : Timed out while waiting for systemd to remove kubepods-burstable-pod7caf6588_e2a5_4f42_b401_2810e464570a.slice" Oct 30 22:22:47.912194 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:22:47.912159 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:23:07.723509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:07.723438 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp"] Oct 30 22:23:07.728990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:07.728966 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:07.782494 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:07.782466 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp"] Oct 30 22:23:07.825334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:07.825295 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a3357\"" Oct 30 22:23:07.825674 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:07.825430 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a3357-dockercfg-452wt\"" Oct 30 22:23:07.909780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:07.909753 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc60c98-0d71-4540-886b-6fe4765d7cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:07.909928 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:07.909821 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6x5\" (UniqueName: \"kubernetes.io/projected/8fc60c98-0d71-4540-886b-6fe4765d7cff-kube-api-access-tf6x5\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:08.010796 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.010768 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc60c98-0d71-4540-886b-6fe4765d7cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:08.010951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.010830 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6x5\" (UniqueName: \"kubernetes.io/projected/8fc60c98-0d71-4540-886b-6fe4765d7cff-kube-api-access-tf6x5\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:08.011478 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.011460 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc60c98-0d71-4540-886b-6fe4765d7cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:08.070141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.070111 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6x5\" (UniqueName: \"kubernetes.io/projected/8fc60c98-0d71-4540-886b-6fe4765d7cff-kube-api-access-tf6x5\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:08.346756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.346671 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:08.532154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.532120 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv"] Oct 30 22:23:08.537209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.537187 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:08.590227 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.590195 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv"] Oct 30 22:23:08.722276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.722170 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:08.722276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.722254 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh428\" (UniqueName: \"kubernetes.io/projected/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-kube-api-access-gh428\") pod \"payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:08.765675 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.765648 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp"] Oct 30 22:23:08.772593 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:23:08.772567 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc60c98_0d71_4540_886b_6fe4765d7cff.slice/crio-f64e869214385a0773e3e3176f3d41672022b7e4f4e59efe10f390bddf8db79b WatchSource:0}: Error finding container f64e869214385a0773e3e3176f3d41672022b7e4f4e59efe10f390bddf8db79b: Status 404 returned error can't find the container with id f64e869214385a0773e3e3176f3d41672022b7e4f4e59efe10f390bddf8db79b Oct 30 22:23:08.825890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.825842 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:08.827746 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.827489 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:08.827904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.827849 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh428\" (UniqueName: \"kubernetes.io/projected/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-kube-api-access-gh428\") pod \"payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:08.886217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:08.886182 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh428\" (UniqueName: \"kubernetes.io/projected/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-kube-api-access-gh428\") pod \"payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:09.061608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.061570 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq"] Oct 30 22:23:09.061929 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.061880 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" podUID="3209dbf8-21b4-4e34-ac01-ced869752247" containerName="payload-processing" containerID="cri-o://6e21152ab93c6177a329a400ed302714e2d755531b1219b70121f48140c8a855" gracePeriod=30 Oct 30 22:23:09.112894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.112857 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j"] Oct 30 22:23:09.113154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.113103 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" podUID="1846b75c-df63-43f5-bbe3-6ce68ac286d6" containerName="payload-pre-processing" containerID="cri-o://6f6ce78df12ff10c2253ff65c2f5231e9044ca87243dfb834d7695922751280e" gracePeriod=30 Oct 30 22:23:09.155550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.155520 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:09.400119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.400077 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" event={"ID":"8fc60c98-0d71-4540-886b-6fe4765d7cff","Type":"ContainerStarted","Data":"d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791"} Oct 30 22:23:09.400265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.400180 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" event={"ID":"8fc60c98-0d71-4540-886b-6fe4765d7cff","Type":"ContainerStarted","Data":"f64e869214385a0773e3e3176f3d41672022b7e4f4e59efe10f390bddf8db79b"} Oct 30 22:23:09.402709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.402165 4991 generic.go:356] "Generic (PLEG): container finished" podID="1846b75c-df63-43f5-bbe3-6ce68ac286d6" containerID="6f6ce78df12ff10c2253ff65c2f5231e9044ca87243dfb834d7695922751280e" exitCode=0 Oct 30 22:23:09.402709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.402241 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" event={"ID":"1846b75c-df63-43f5-bbe3-6ce68ac286d6","Type":"ContainerDied","Data":"6f6ce78df12ff10c2253ff65c2f5231e9044ca87243dfb834d7695922751280e"} Oct 30 22:23:09.405480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.405449 4991 generic.go:356] "Generic (PLEG): container finished" podID="3209dbf8-21b4-4e34-ac01-ced869752247" containerID="6e21152ab93c6177a329a400ed302714e2d755531b1219b70121f48140c8a855" exitCode=0 Oct 30 22:23:09.405590 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.405505 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" event={"ID":"3209dbf8-21b4-4e34-ac01-ced869752247","Type":"ContainerDied","Data":"6e21152ab93c6177a329a400ed302714e2d755531b1219b70121f48140c8a855"} Oct 30 22:23:09.484789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.484764 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:23:09.488372 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.488354 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:23:09.516096 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.516056 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" podStartSLOduration=2.5160433920000003 podStartE2EDuration="2.516043392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:09.512532142 +0000 UTC m=+2209.811942044" watchObservedRunningTime="2026-10-30 22:23:09.516043392 +0000 UTC m=+2209.815453272" Oct 30 22:23:09.598651 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.598623 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv"] Oct 30 22:23:09.604757 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:23:09.604727 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4798bcb_37da_4f81_aad9_0ef7ac6f4372.slice/crio-48ed88ef7322f3b2784a36528815f77a6958b79a3a65ca13100d272b01365957 WatchSource:0}: Error finding container 48ed88ef7322f3b2784a36528815f77a6958b79a3a65ca13100d272b01365957: Status 404 returned error can't find the container with id 48ed88ef7322f3b2784a36528815f77a6958b79a3a65ca13100d272b01365957 Oct 30 22:23:09.649211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.647591 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1846b75c-df63-43f5-bbe3-6ce68ac286d6-kube-api-access-k7c2x\" (UniqueName: \"kubernetes.io/projected/1846b75c-df63-43f5-bbe3-6ce68ac286d6-kube-api-access-k7c2x\") pod \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " Oct 30 22:23:09.649211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.648653 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1846b75c-df63-43f5-bbe3-6ce68ac286d6-plugins-config-volume" pod "1846b75c-df63-43f5-bbe3-6ce68ac286d6" (UID: "1846b75c-df63-43f5-bbe3-6ce68ac286d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:09.649211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.648722 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1846b75c-df63-43f5-bbe3-6ce68ac286d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1846b75c-df63-43f5-bbe3-6ce68ac286d6-plugins-config-volume\") pod \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\" (UID: \"1846b75c-df63-43f5-bbe3-6ce68ac286d6\") " Oct 30 22:23:09.649211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.648901 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3209dbf8-21b4-4e34-ac01-ced869752247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3209dbf8-21b4-4e34-ac01-ced869752247-plugins-config-volume\") pod \"3209dbf8-21b4-4e34-ac01-ced869752247\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " Oct 30 22:23:09.649211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.648957 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3209dbf8-21b4-4e34-ac01-ced869752247-kube-api-access-8k5tz\" (UniqueName: \"kubernetes.io/projected/3209dbf8-21b4-4e34-ac01-ced869752247-kube-api-access-8k5tz\") pod \"3209dbf8-21b4-4e34-ac01-ced869752247\" (UID: \"3209dbf8-21b4-4e34-ac01-ced869752247\") " Oct 30 22:23:09.652140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.650276 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3209dbf8-21b4-4e34-ac01-ced869752247-plugins-config-volume" pod "3209dbf8-21b4-4e34-ac01-ced869752247" (UID: "3209dbf8-21b4-4e34-ac01-ced869752247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:09.652140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.651398 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1846b75c-df63-43f5-bbe3-6ce68ac286d6-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:09.657387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.654927 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3209dbf8-21b4-4e34-ac01-ced869752247-kube-api-access-8k5tz" pod "3209dbf8-21b4-4e34-ac01-ced869752247" (UID: "3209dbf8-21b4-4e34-ac01-ced869752247"). InnerVolumeSpecName "kube-api-access-8k5tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:09.682340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.678493 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1846b75c-df63-43f5-bbe3-6ce68ac286d6-kube-api-access-k7c2x" pod "1846b75c-df63-43f5-bbe3-6ce68ac286d6" (UID: "1846b75c-df63-43f5-bbe3-6ce68ac286d6"). InnerVolumeSpecName "kube-api-access-k7c2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:09.759022 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.758311 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7c2x\" (UniqueName: \"kubernetes.io/projected/1846b75c-df63-43f5-bbe3-6ce68ac286d6-kube-api-access-k7c2x\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:09.759022 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.758633 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3209dbf8-21b4-4e34-ac01-ced869752247-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:09.759022 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:09.758651 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5tz\" (UniqueName: \"kubernetes.io/projected/3209dbf8-21b4-4e34-ac01-ced869752247-kube-api-access-8k5tz\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:10.413908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.413872 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" event={"ID":"b4798bcb-37da-4f81-aad9-0ef7ac6f4372","Type":"ContainerStarted","Data":"30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340"} Oct 30 22:23:10.414382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.413938 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:10.414382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.413959 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" event={"ID":"b4798bcb-37da-4f81-aad9-0ef7ac6f4372","Type":"ContainerStarted","Data":"48ed88ef7322f3b2784a36528815f77a6958b79a3a65ca13100d272b01365957"} Oct 30 22:23:10.415464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.415443 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" Oct 30 22:23:10.415589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.415472 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j" event={"ID":"1846b75c-df63-43f5-bbe3-6ce68ac286d6","Type":"ContainerDied","Data":"b4ae75705800aa5236628afad2044b340d93254aa4802cfe93e1b41bcf31bc60"} Oct 30 22:23:10.415589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.415528 4991 scope.go:122] "RemoveContainer" containerID="6f6ce78df12ff10c2253ff65c2f5231e9044ca87243dfb834d7695922751280e" Oct 30 22:23:10.416867 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.416847 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" Oct 30 22:23:10.416977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.416901 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq" event={"ID":"3209dbf8-21b4-4e34-ac01-ced869752247","Type":"ContainerDied","Data":"feef1c0e5c58ceae44499d7a21660215cbf8b191138c3efbc30ba5f7dd420c64"} Oct 30 22:23:10.417260 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.417224 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:10.425614 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.425593 4991 scope.go:122] "RemoveContainer" containerID="6e21152ab93c6177a329a400ed302714e2d755531b1219b70121f48140c8a855" Oct 30 22:23:10.527414 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.527356 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" podStartSLOduration=2.527339104 podStartE2EDuration="2.527339104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:10.525175014 +0000 UTC m=+2210.824584896" watchObservedRunningTime="2026-10-30 22:23:10.527339104 +0000 UTC m=+2210.826748985" Oct 30 22:23:10.633680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.633649 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j"] Oct 30 22:23:10.637214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.637190 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-66dd4b5799-bt65j"] Oct 30 22:23:10.773961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.773416 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq"] Oct 30 22:23:10.781038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:10.780995 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-6cbcc5665b-2ghgq"] Oct 30 22:23:11.874583 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:11.874553 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1846b75c-df63-43f5-bbe3-6ce68ac286d6" path="/var/lib/kubelet/pods/1846b75c-df63-43f5-bbe3-6ce68ac286d6/volumes" Oct 30 22:23:11.874969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:11.874955 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3209dbf8-21b4-4e34-ac01-ced869752247" path="/var/lib/kubelet/pods/3209dbf8-21b4-4e34-ac01-ced869752247/volumes" Oct 30 22:23:17.921191 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:23:17.921146 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:23:21.428913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:21.428883 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:23:21.429338 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:21.428966 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:23:28.092199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:28.092167 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv_b4798bcb-37da-4f81-aad9-0ef7ac6f4372/payload-processing/0.log" Oct 30 22:23:39.903965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.903931 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj"] Oct 30 22:23:39.904498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.904483 4991 cpu_manager.go:414] "containerMap: removing container" podUID="3209dbf8-21b4-4e34-ac01-ced869752247" containerName="payload-processing" Oct 30 22:23:39.904557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.904500 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3209dbf8-21b4-4e34-ac01-ced869752247" containerName="payload-processing" Oct 30 22:23:39.904557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.904515 4991 cpu_manager.go:414] "containerMap: removing container" podUID="1846b75c-df63-43f5-bbe3-6ce68ac286d6" containerName="payload-pre-processing" Oct 30 22:23:39.904557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.904520 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1846b75c-df63-43f5-bbe3-6ce68ac286d6" containerName="payload-pre-processing" Oct 30 22:23:39.904657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.904607 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3209dbf8-21b4-4e34-ac01-ced869752247" containerName="payload-processing" Oct 30 22:23:39.904657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.904618 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1846b75c-df63-43f5-bbe3-6ce68ac286d6" containerName="payload-pre-processing" Oct 30 22:23:39.909635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.909613 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:39.959896 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.959872 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ebcc6d-serving-cert\"" Oct 30 22:23:39.990782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.990754 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbtw\" (UniqueName: \"kubernetes.io/projected/4efc4ee7-85a6-4338-aad3-5735bbea322e-kube-api-access-ltbtw\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:39.990908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.990800 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:39.990908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.990897 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-metrics-certs\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:39.991069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:39.991048 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efc4ee7-85a6-4338-aad3-5735bbea322e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.009863 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.009840 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj"] Oct 30 22:23:40.092272 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.092237 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efc4ee7-85a6-4338-aad3-5735bbea322e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.092478 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.092312 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbtw\" (UniqueName: \"kubernetes.io/projected/4efc4ee7-85a6-4338-aad3-5735bbea322e-kube-api-access-ltbtw\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.092478 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.092369 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.092478 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.092409 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-metrics-certs\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.092651 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:23:40.092507 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-serving-cert: secret "maas-api-e2e-praxis-b-ebcc6d-serving-cert" not found Oct 30 22:23:40.092651 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:23:40.092577 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls podName:4efc4ee7-85a6-4338-aad3-5735bbea322e nodeName:}" failed. No retries permitted until 2026-10-30 22:23:40.592562747 +0000 UTC m=+2240.891972618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls") pod "maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" (UID: "4efc4ee7-85a6-4338-aad3-5735bbea322e") : secret "maas-api-e2e-praxis-b-ebcc6d-serving-cert" not found Oct 30 22:23:40.092908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.092889 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efc4ee7-85a6-4338-aad3-5735bbea322e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.094721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.094700 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-metrics-certs\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.159113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.158263 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbtw\" (UniqueName: \"kubernetes.io/projected/4efc4ee7-85a6-4338-aad3-5735bbea322e-kube-api-access-ltbtw\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.599766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.599727 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.602269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.602238 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:40.823074 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:40.823041 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:41.213552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:41.213525 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj"] Oct 30 22:23:41.215370 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:23:41.215310 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efc4ee7_85a6_4338_aad3_5735bbea322e.slice/crio-85bc4027f0204f689b123f048914ba1252a47ef624e3d84d588f2a67bdcf143d WatchSource:0}: Error finding container 85bc4027f0204f689b123f048914ba1252a47ef624e3d84d588f2a67bdcf143d: Status 404 returned error can't find the container with id 85bc4027f0204f689b123f048914ba1252a47ef624e3d84d588f2a67bdcf143d Oct 30 22:23:41.642832 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:41.642787 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" event={"ID":"4efc4ee7-85a6-4338-aad3-5735bbea322e","Type":"ContainerStarted","Data":"85bc4027f0204f689b123f048914ba1252a47ef624e3d84d588f2a67bdcf143d"} Oct 30 22:23:42.650118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:42.650083 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" event={"ID":"4efc4ee7-85a6-4338-aad3-5735bbea322e","Type":"ContainerStarted","Data":"6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48"} Oct 30 22:23:42.650526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:42.650168 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:23:42.763144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:42.763082 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" podStartSLOduration=2.835129224 podStartE2EDuration="3.763063784s" totalImagesPullingTime="927.93456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:42.762775584 +0000 UTC m=+2243.062185486" watchObservedRunningTime="2026-10-30 22:23:42.763063784 +0000 UTC m=+2243.062473665" Oct 30 22:23:47.910762 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:23:47.910725 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:23:48.663233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:23:48.663193 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:24:17.911774 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:24:17.911734 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:24:25.767496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.767463 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf"] Oct 30 22:24:25.772109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.772083 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:25.825962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.825925 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:25.826121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.826089 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzct\" (UniqueName: \"kubernetes.io/projected/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-kube-api-access-xxzct\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:25.869690 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.869656 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-872f4a-dockercfg-rlgzf\"" Oct 30 22:24:25.869867 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.869695 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-872f4a\"" Oct 30 22:24:25.875805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.875779 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf"] Oct 30 22:24:25.927057 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.927030 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:25.927195 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.927130 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzct\" (UniqueName: \"kubernetes.io/projected/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-kube-api-access-xxzct\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:25.927790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.927767 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:25.986740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:25.986714 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzct\" (UniqueName: \"kubernetes.io/projected/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-kube-api-access-xxzct\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:26.086840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.086780 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:26.465872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.465789 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf"] Oct 30 22:24:26.472027 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:24:26.471996 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e6dafd_7a9b_4246_bcb4_b64cc711332c.slice/crio-b202d3d77aa505e32337dbadaaab159fa68e862635e6471de7af7021153f61aa WatchSource:0}: Error finding container b202d3d77aa505e32337dbadaaab159fa68e862635e6471de7af7021153f61aa: Status 404 returned error can't find the container with id b202d3d77aa505e32337dbadaaab159fa68e862635e6471de7af7021153f61aa Oct 30 22:24:26.558600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.558578 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm"] Oct 30 22:24:26.562627 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.562609 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.621984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.621954 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm"] Oct 30 22:24:26.636895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.636862 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n562l\" (UniqueName: \"kubernetes.io/projected/7b198550-6a6d-42d1-b34b-251b9921ea49-kube-api-access-n562l\") pod \"payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.637023 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.636993 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b198550-6a6d-42d1-b34b-251b9921ea49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.738607 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.738577 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n562l\" (UniqueName: \"kubernetes.io/projected/7b198550-6a6d-42d1-b34b-251b9921ea49-kube-api-access-n562l\") pod \"payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.738757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.738626 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b198550-6a6d-42d1-b34b-251b9921ea49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.739291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.739273 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b198550-6a6d-42d1-b34b-251b9921ea49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.801612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.801589 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n562l\" (UniqueName: \"kubernetes.io/projected/7b198550-6a6d-42d1-b34b-251b9921ea49-kube-api-access-n562l\") pod \"payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.878456 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.878423 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:26.956010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.955978 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" event={"ID":"a9e6dafd-7a9b-4246-bcb4-b64cc711332c","Type":"ContainerStarted","Data":"dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7"} Oct 30 22:24:26.956198 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.956183 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" event={"ID":"a9e6dafd-7a9b-4246-bcb4-b64cc711332c","Type":"ContainerStarted","Data":"b202d3d77aa505e32337dbadaaab159fa68e862635e6471de7af7021153f61aa"} Oct 30 22:24:26.957188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:26.957166 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:27.076927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:27.076836 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" podStartSLOduration=2.07681902 podStartE2EDuration="2.07681902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:27.070221753 +0000 UTC m=+2287.369631653" watchObservedRunningTime="2026-10-30 22:24:27.07681902 +0000 UTC m=+2287.376228902" Oct 30 22:24:27.266565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:27.266529 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm"] Oct 30 22:24:27.276371 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:24:27.276308 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b198550_6a6d_42d1_b34b_251b9921ea49.slice/crio-12a822e6277477d1aa600fe1d669b8bc8964219e47f2a2621a637164e7add48d WatchSource:0}: Error finding container 12a822e6277477d1aa600fe1d669b8bc8964219e47f2a2621a637164e7add48d: Status 404 returned error can't find the container with id 12a822e6277477d1aa600fe1d669b8bc8964219e47f2a2621a637164e7add48d Oct 30 22:24:27.966026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:27.965989 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" event={"ID":"7b198550-6a6d-42d1-b34b-251b9921ea49","Type":"ContainerStarted","Data":"beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830"} Oct 30 22:24:27.966509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:27.966051 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" event={"ID":"7b198550-6a6d-42d1-b34b-251b9921ea49","Type":"ContainerStarted","Data":"12a822e6277477d1aa600fe1d669b8bc8964219e47f2a2621a637164e7add48d"} Oct 30 22:24:27.966509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:27.966277 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:38.975616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:38.975585 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:24:38.976113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:38.975665 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:24:39.087751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:39.087704 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" podStartSLOduration=13.08769034 podStartE2EDuration="13.08769034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:28.078976335 +0000 UTC m=+2288.378386222" watchObservedRunningTime="2026-10-30 22:24:39.08769034 +0000 UTC m=+2299.387100219" Oct 30 22:24:45.221379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.221301 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv"] Oct 30 22:24:45.226242 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.226222 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.276608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.276573 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c89e47-serving-cert\"" Oct 30 22:24:45.329510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.329469 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv"] Oct 30 22:24:45.347529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.347287 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxfl\" (UniqueName: \"kubernetes.io/projected/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-kube-api-access-shxfl\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.347529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.347375 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.347529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.347425 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.347529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.347482 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.448638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.448597 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxfl\" (UniqueName: \"kubernetes.io/projected/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-kube-api-access-shxfl\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.448814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.448668 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.448814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.448698 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.448814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.448733 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.449638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.449610 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.451297 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.451275 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.451415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.451302 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.506268 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.506238 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxfl\" (UniqueName: \"kubernetes.io/projected/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-kube-api-access-shxfl\") pod \"maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.540528 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.540502 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:45.917837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:45.917761 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv"] Oct 30 22:24:45.923596 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:24:45.923571 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8397b25f_e3c5_420e_bcd8_e4c1181a49dc.slice/crio-6e50db6b35a52200a6a4256cb134d83068e9ef35c63c73c6db92c56ef6a35676 WatchSource:0}: Error finding container 6e50db6b35a52200a6a4256cb134d83068e9ef35c63c73c6db92c56ef6a35676: Status 404 returned error can't find the container with id 6e50db6b35a52200a6a4256cb134d83068e9ef35c63c73c6db92c56ef6a35676 Oct 30 22:24:46.107766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:46.107729 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" event={"ID":"8397b25f-e3c5-420e-bcd8-e4c1181a49dc","Type":"ContainerStarted","Data":"6e50db6b35a52200a6a4256cb134d83068e9ef35c63c73c6db92c56ef6a35676"} Oct 30 22:24:47.116592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:47.116558 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" event={"ID":"8397b25f-e3c5-420e-bcd8-e4c1181a49dc","Type":"ContainerStarted","Data":"0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113"} Oct 30 22:24:47.116996 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:47.116621 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:47.271523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:47.270943 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" podStartSLOduration=1.291215496 podStartE2EDuration="2.270923945s" totalImagesPullingTime="979.708449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:47.268622145 +0000 UTC m=+2307.568032027" watchObservedRunningTime="2026-10-30 22:24:47.270923945 +0000 UTC m=+2307.570333826" Oct 30 22:24:47.911925 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:24:47.911878 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:24:51.277061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.277024 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz"] Oct 30 22:24:51.281815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.281797 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.317693 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.317659 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.317816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.317705 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.317816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.317759 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.317903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.317889 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.317958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.317907 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wng\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-kube-api-access-25wng\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.318013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.317984 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.318069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.318039 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.318127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.318070 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.318247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.318233 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.330814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.330787 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ae8879-openshift-default-dockercfg-j62zp\"" Oct 30 22:24:51.419682 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.419631 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.419884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.419712 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.419956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.419871 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420023 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.419995 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.420020 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wng\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-kube-api-access-25wng\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.420054 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.420090 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.420116 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420310 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.420263 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420484 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.420459 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.420682 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.420660 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.421296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.421271 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.421673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.421651 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.421945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.421903 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.425156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.425135 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.427770 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.427749 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.480575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.479858 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz"] Oct 30 22:24:51.482149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.482122 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.482290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.482271 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wng\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-kube-api-access-25wng\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:51.597509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:51.597436 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:52.036481 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.036447 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz"] Oct 30 22:24:52.042827 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:24:52.042802 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9c61f6_a801_4f19_8d90_44cf2d436c60.slice/crio-146d9eed289204d3fc0aa3c14aa310b25e984d6cad1c5642dbd446e37d5552e0 WatchSource:0}: Error finding container 146d9eed289204d3fc0aa3c14aa310b25e984d6cad1c5642dbd446e37d5552e0: Status 404 returned error can't find the container with id 146d9eed289204d3fc0aa3c14aa310b25e984d6cad1c5642dbd446e37d5552e0 Oct 30 22:24:52.045281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.045250 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:52.045417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.045332 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:52.045417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.045369 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:24:52.178928 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.178891 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" event={"ID":"ef9c61f6-a801-4f19-8d90-44cf2d436c60","Type":"ContainerStarted","Data":"9c392279c9584f67d0c600ed2a35ff4b313d0f2ae53a06d1d0dee7d92522c348"} Oct 30 22:24:52.179083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.178975 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" event={"ID":"ef9c61f6-a801-4f19-8d90-44cf2d436c60","Type":"ContainerStarted","Data":"146d9eed289204d3fc0aa3c14aa310b25e984d6cad1c5642dbd446e37d5552e0"} Oct 30 22:24:52.294911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.294806 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" podStartSLOduration=1.294787948 podStartE2EDuration="1.294787948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:52.292863511 +0000 UTC m=+2312.592273411" watchObservedRunningTime="2026-10-30 22:24:52.294787948 +0000 UTC m=+2312.594197830" Oct 30 22:24:52.598627 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:52.598548 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:53.156335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:53.156296 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:24:53.382741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:53.382707 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz_fd453ded-f246-4c99-aca7-a2a4d3434eb3/payload-processing/0.log" Oct 30 22:24:53.599755 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:53.599718 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:24:53.599963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:53.599775 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:24:54.155020 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:54.154989 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv_b4798bcb-37da-4f81-aad9-0ef7ac6f4372/payload-processing/0.log" Oct 30 22:24:54.382683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:54.382654 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv_b4798bcb-37da-4f81-aad9-0ef7ac6f4372/payload-processing/0.log" Oct 30 22:24:54.598781 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:54.598746 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:24:54.599150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:54.598800 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:24:55.598238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:55.598205 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:24:55.598427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:55.598254 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:24:56.598602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:56.598567 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:24:56.599005 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:56.598634 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:24:57.598254 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:57.598219 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:24:57.598460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:57.598284 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 22:24:58.603117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:58.603087 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:58.603633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:58.603618 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:24:58.604370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:24:58.604346 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:25:17.911960 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:25:17.911920 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:25:38.178603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.178571 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv"] Oct 30 22:25:38.179147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.178766 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" containerID="cri-o://fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38" gracePeriod=30 Oct 30 22:25:38.236419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.236381 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787"] Oct 30 22:25:38.236999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.236958 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" containerID="cri-o://9566873e46f70d9bae1a8cd977183d22dc753ede94d28f6828d742e442214e1d" gracePeriod=30 Oct 30 22:25:38.237987 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.237962 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7"] Oct 30 22:25:38.238580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.238529 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" containerID="cri-o://bb15d84f712e6196d9a3d460d326a91ac9866989a5d346cd4950a9705248e4a1" gracePeriod=30 Oct 30 22:25:38.274861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.274814 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2"] Oct 30 22:25:38.275082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:38.275040 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" containerID="cri-o://cd5c8217de1bf3fd6c6d98ad40594c5256ef5ed0458226e51c7f9736ba88aeac" gracePeriod=30 Oct 30 22:25:41.726086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:41.726052 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54"] Oct 30 22:25:41.726493 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:41.726264 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" containerID="cri-o://81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526" gracePeriod=30 Oct 30 22:25:43.545525 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.545498 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:25:43.585288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.585192 4991 generic.go:356] "Generic (PLEG): container finished" podID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerID="cd5c8217de1bf3fd6c6d98ad40594c5256ef5ed0458226e51c7f9736ba88aeac" exitCode=0 Oct 30 22:25:43.585288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.585241 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" event={"ID":"31b2c50f-940b-4d9c-9c27-4f83b7667a78","Type":"ContainerDied","Data":"cd5c8217de1bf3fd6c6d98ad40594c5256ef5ed0458226e51c7f9736ba88aeac"} Oct 30 22:25:43.588497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.587952 4991 generic.go:356] "Generic (PLEG): container finished" podID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerID="bb15d84f712e6196d9a3d460d326a91ac9866989a5d346cd4950a9705248e4a1" exitCode=0 Oct 30 22:25:43.588497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.588053 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" event={"ID":"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd","Type":"ContainerDied","Data":"bb15d84f712e6196d9a3d460d326a91ac9866989a5d346cd4950a9705248e4a1"} Oct 30 22:25:43.592610 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.592497 4991 generic.go:356] "Generic (PLEG): container finished" podID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerID="9566873e46f70d9bae1a8cd977183d22dc753ede94d28f6828d742e442214e1d" exitCode=0 Oct 30 22:25:43.592610 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.592524 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" event={"ID":"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e","Type":"ContainerDied","Data":"9566873e46f70d9bae1a8cd977183d22dc753ede94d28f6828d742e442214e1d"} Oct 30 22:25:43.594498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.594474 4991 generic.go:356] "Generic (PLEG): container finished" podID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerID="fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38" exitCode=0 Oct 30 22:25:43.594620 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.594530 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" event={"ID":"37ac1480-8a43-404b-9787-ed5d05d9427d","Type":"ContainerDied","Data":"fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38"} Oct 30 22:25:43.594620 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.594570 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" event={"ID":"37ac1480-8a43-404b-9787-ed5d05d9427d","Type":"ContainerDied","Data":"0de5bfbaffd9ad7d17e27edf7fe2b543f673b667b488271d8896c05679b46efb"} Oct 30 22:25:43.594620 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.594599 4991 scope.go:122] "RemoveContainer" containerID="fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38" Oct 30 22:25:43.594865 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.594847 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv" Oct 30 22:25:43.605138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605112 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ac1480-8a43-404b-9787-ed5d05d9427d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac1480-8a43-404b-9787-ed5d05d9427d-istiod-ca-cert\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605180 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-socket\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605380 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-data\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605430 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-credential-socket\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605502 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605478 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-podinfo\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605513 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-token\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-token\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605543 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-certs\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605617 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605576 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-kube-api-access-9fxxx\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-kube-api-access-9fxxx\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.605652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.605633 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-envoy\") pod \"37ac1480-8a43-404b-9787-ed5d05d9427d\" (UID: \"37ac1480-8a43-404b-9787-ed5d05d9427d\") " Oct 30 22:25:43.607234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.606683 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-socket" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.607234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.606820 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-data" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.607234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.607072 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-certs" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.607234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.607217 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ac1480-8a43-404b-9787-ed5d05d9427d-istiod-ca-cert" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:43.607497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.607385 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-credential-socket" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.611004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.610983 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-kube-api-access-9fxxx" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "kube-api-access-9fxxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.612844 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.612824 4991 scope.go:122] "RemoveContainer" containerID="fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38" Oct 30 22:25:43.613220 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:25:43.613152 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38\": container with ID starting with fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38 not found: ID does not exist" containerID="fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38" Oct 30 22:25:43.613220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.613193 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38"} err="failed to get container status \"fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38\": rpc error: code = NotFound desc = could not find container \"fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38\": container with ID starting with fba76c3a8bc349acd098329cc685aab1bf00f76a414092d34f83bd6254366c38 not found: ID does not exist" Oct 30 22:25:43.615497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.615473 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-envoy" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.615497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.615485 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-podinfo" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:43.617370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.617304 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-token" pod "37ac1480-8a43-404b-9787-ed5d05d9427d" (UID: "37ac1480-8a43-404b-9787-ed5d05d9427d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.648279 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.648252 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:25:43.651953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.651930 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:25:43.656010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.655993 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:25:43.712419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712379 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-certs\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.712586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712439 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istiod-ca-cert\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.712586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712472 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-certs\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.712586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712512 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istiod-ca-cert\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.712586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712578 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-kube-api-access-fbcnw\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-kube-api-access-fbcnw\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712604 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-socket\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712635 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istiod-ca-cert\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712676 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-envoy\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712750 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-token\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-token\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712799 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-token\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712834 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-data\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712878 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-podinfo\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.712917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712907 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-podinfo\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712951 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-kube-api-access-4n5lg\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-kube-api-access-4n5lg\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.712999 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-credential-socket\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713052 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-credential-socket\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713086 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-data\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713127 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-envoy\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713173 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-podinfo\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713239 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-data\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713276 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-kube-api-access-5rk6s\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-kube-api-access-5rk6s\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713302 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-socket\") pod \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\" (UID: \"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713354 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-certs\") pod \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\" (UID: \"31b2c50f-940b-4d9c-9c27-4f83b7667a78\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713412 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-envoy\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713456 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-credential-socket\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713488 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-token\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-token\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713522 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-socket\") pod \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\" (UID: \"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e\") " Oct 30 22:25:43.714097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.713867 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-data" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.714930 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.714163 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-socket" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.714930 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.714357 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-certs" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.714930 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.714568 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-data" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.714930 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.714807 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istiod-ca-cert" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:43.715180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.714957 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-socket" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.715383 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.714974 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-credential-socket" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715762 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715806 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac1480-8a43-404b-9787-ed5d05d9427d-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715826 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715842 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715857 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715876 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715893 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715908 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715924 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715944 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715959 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac1480-8a43-404b-9787-ed5d05d9427d-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.715973 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fxxx\" (UniqueName: \"kubernetes.io/projected/37ac1480-8a43-404b-9787-ed5d05d9427d-kube-api-access-9fxxx\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.716937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.716333 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-certs" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.716937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.716414 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-certs" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.716937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.716637 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istiod-ca-cert" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:43.716937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.716660 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-credential-socket" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.716937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.716704 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-socket" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.717830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.717792 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istiod-ca-cert" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:43.718285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.718247 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-data" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.718435 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.718399 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-credential-socket" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.718524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.718497 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-podinfo" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:43.719362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.719242 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-kube-api-access-4n5lg" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "kube-api-access-4n5lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.719698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.719566 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-kube-api-access-fbcnw" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "kube-api-access-fbcnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.719698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.719648 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-token" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.720763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.720739 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-envoy" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.722716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.722690 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-kube-api-access-5rk6s" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "kube-api-access-5rk6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.722818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.722759 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-envoy" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.722886 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.722829 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-podinfo" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:43.723925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.723056 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-token" pod "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" (UID: "1fe4e4a8-11eb-4720-bfac-a62d1136d8bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.723925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.723280 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-podinfo" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:43.723925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.723423 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-envoy" pod "31b2c50f-940b-4d9c-9c27-4f83b7667a78" (UID: "31b2c50f-940b-4d9c-9c27-4f83b7667a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.724571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.724548 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-token" pod "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" (UID: "e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.817042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.816978 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817004 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817013 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817022 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbcnw\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-kube-api-access-fbcnw\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817032 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817042 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817050 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817060 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817068 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817077 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817086 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817094 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817102 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5lg\" (UniqueName: \"kubernetes.io/projected/31b2c50f-940b-4d9c-9c27-4f83b7667a78-kube-api-access-4n5lg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817110 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817118 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817126 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817135 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817143 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b2c50f-940b-4d9c-9c27-4f83b7667a78-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817151 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rk6s\" (UniqueName: \"kubernetes.io/projected/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-kube-api-access-5rk6s\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817159 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817168 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b2c50f-940b-4d9c-9c27-4f83b7667a78-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817177 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817186 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.817336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:43.817195 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.290534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.290501 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv"] Oct 30 22:25:44.302485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.302450 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-7hsfv"] Oct 30 22:25:44.602468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.602433 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" Oct 30 22:25:44.602918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.602432 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2" event={"ID":"31b2c50f-940b-4d9c-9c27-4f83b7667a78","Type":"ContainerDied","Data":"f638f6c8d961870627701b796435bb34ba649a7418806b75dadf3e9a16ec5a0e"} Oct 30 22:25:44.602918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.602582 4991 scope.go:122] "RemoveContainer" containerID="cd5c8217de1bf3fd6c6d98ad40594c5256ef5ed0458226e51c7f9736ba88aeac" Oct 30 22:25:44.604366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.604105 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" event={"ID":"1fe4e4a8-11eb-4720-bfac-a62d1136d8bd","Type":"ContainerDied","Data":"07d304b3171b68cea9fc8845bf3a190d9d5dc5b1180eae2ddd8e4a2f6f0cecf8"} Oct 30 22:25:44.604366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.604189 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7" Oct 30 22:25:44.605785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.605761 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" event={"ID":"e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e","Type":"ContainerDied","Data":"085158baadc669ab7a94173d74b640b6c2c639de2c2298faad213babd33a2c47"} Oct 30 22:25:44.605901 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.605866 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787" Oct 30 22:25:44.614162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.614146 4991 scope.go:122] "RemoveContainer" containerID="bb15d84f712e6196d9a3d460d326a91ac9866989a5d346cd4950a9705248e4a1" Oct 30 22:25:44.623061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.623038 4991 scope.go:122] "RemoveContainer" containerID="9566873e46f70d9bae1a8cd977183d22dc753ede94d28f6828d742e442214e1d" Oct 30 22:25:44.762737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.762703 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7"] Oct 30 22:25:44.768763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.768734 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-sxjs7"] Oct 30 22:25:44.870540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.870465 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2"] Oct 30 22:25:44.874362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.874315 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-f84b2"] Oct 30 22:25:44.980546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.980388 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787"] Oct 30 22:25:44.983507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:44.983482 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gz787"] Oct 30 22:25:45.876274 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:45.876237 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" path="/var/lib/kubelet/pods/1fe4e4a8-11eb-4720-bfac-a62d1136d8bd/volumes" Oct 30 22:25:45.876764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:45.876748 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" path="/var/lib/kubelet/pods/31b2c50f-940b-4d9c-9c27-4f83b7667a78/volumes" Oct 30 22:25:45.877181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:45.877168 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" path="/var/lib/kubelet/pods/37ac1480-8a43-404b-9787-ed5d05d9427d/volumes" Oct 30 22:25:45.877791 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:45.877774 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" path="/var/lib/kubelet/pods/e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e/volumes" Oct 30 22:25:46.710340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:46.706949 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j"] Oct 30 22:25:46.710948 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:46.710909 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" podUID="de363edf-a0ea-4b46-b80c-53157e82dbf5" containerName="maas-api" containerID="cri-o://645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c" gracePeriod=30 Oct 30 22:25:47.071562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.070545 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:25:47.115765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.115735 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:25:47.163545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163514 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-data\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.163726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163563 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-kube-api-access-fvbp5\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-kube-api-access-fvbp5\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.163726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163581 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-token\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.163726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163616 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de363edf-a0ea-4b46-b80c-53157e82dbf5-kube-api-access-pbj78\" (UniqueName: \"kubernetes.io/projected/de363edf-a0ea-4b46-b80c-53157e82dbf5-kube-api-access-pbj78\") pod \"de363edf-a0ea-4b46-b80c-53157e82dbf5\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " Oct 30 22:25:47.163726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163673 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-metrics-certs\") pod \"de363edf-a0ea-4b46-b80c-53157e82dbf5\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " Oct 30 22:25:47.163962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163727 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-envoy\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.163962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163773 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-socket\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.163962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163801 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-credential-socket\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.163962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163877 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-certs\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.163962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163906 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de363edf-a0ea-4b46-b80c-53157e82dbf5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de363edf-a0ea-4b46-b80c-53157e82dbf5-openshift-service-ca\") pod \"de363edf-a0ea-4b46-b80c-53157e82dbf5\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " Oct 30 22:25:47.163962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163945 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-podinfo\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.164248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164000 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls\") pod \"de363edf-a0ea-4b46-b80c-53157e82dbf5\" (UID: \"de363edf-a0ea-4b46-b80c-53157e82dbf5\") " Oct 30 22:25:47.164248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164069 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istiod-ca-cert\") pod \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\" (UID: \"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1\") " Oct 30 22:25:47.164248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.163796 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-data" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:47.164248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164040 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-socket" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:47.164248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164070 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-credential-socket" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:47.164248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164190 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-certs" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:47.164638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164596 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.164638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164623 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.164777 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164642 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.164777 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164658 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.165016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.164988 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de363edf-a0ea-4b46-b80c-53157e82dbf5-openshift-service-ca" pod "de363edf-a0ea-4b46-b80c-53157e82dbf5" (UID: "de363edf-a0ea-4b46-b80c-53157e82dbf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:47.165200 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.165043 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istiod-ca-cert" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:47.166562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.166537 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-kube-api-access-fvbp5" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "kube-api-access-fvbp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:47.166562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.166551 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-metrics-certs" pod "de363edf-a0ea-4b46-b80c-53157e82dbf5" (UID: "de363edf-a0ea-4b46-b80c-53157e82dbf5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:47.166967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.166943 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-podinfo" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:47.167947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.167922 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-token" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:47.168089 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.168070 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-envoy" pod "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" (UID: "54adce1e-7ecb-40cb-9d22-8ee7fde39dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:47.168450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.168427 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de363edf-a0ea-4b46-b80c-53157e82dbf5-kube-api-access-pbj78" pod "de363edf-a0ea-4b46-b80c-53157e82dbf5" (UID: "de363edf-a0ea-4b46-b80c-53157e82dbf5"). InnerVolumeSpecName "kube-api-access-pbj78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:47.168536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.168519 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls" pod "de363edf-a0ea-4b46-b80c-53157e82dbf5" (UID: "de363edf-a0ea-4b46-b80c-53157e82dbf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:47.265589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265542 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de363edf-a0ea-4b46-b80c-53157e82dbf5-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265584 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265594 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265603 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265612 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvbp5\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-kube-api-access-fvbp5\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265624 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265633 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbj78\" (UniqueName: \"kubernetes.io/projected/de363edf-a0ea-4b46-b80c-53157e82dbf5-kube-api-access-pbj78\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265642 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de363edf-a0ea-4b46-b80c-53157e82dbf5-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.265869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.265651 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.643426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.643332 4991 generic.go:356] "Generic (PLEG): container finished" podID="de363edf-a0ea-4b46-b80c-53157e82dbf5" containerID="645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c" exitCode=0 Oct 30 22:25:47.643426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.643372 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" event={"ID":"de363edf-a0ea-4b46-b80c-53157e82dbf5","Type":"ContainerDied","Data":"645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c"} Oct 30 22:25:47.643426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.643394 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" Oct 30 22:25:47.643697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.643435 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j" event={"ID":"de363edf-a0ea-4b46-b80c-53157e82dbf5","Type":"ContainerDied","Data":"3cf4f4063743c67a3b7dacdbaec6668332fab5153c0f110b4090ce1764804c86"} Oct 30 22:25:47.643697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.643463 4991 scope.go:122] "RemoveContainer" containerID="645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c" Oct 30 22:25:47.645211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.645186 4991 generic.go:356] "Generic (PLEG): container finished" podID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerID="81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526" exitCode=0 Oct 30 22:25:47.645334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.645219 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" event={"ID":"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1","Type":"ContainerDied","Data":"81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526"} Oct 30 22:25:47.645334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.645270 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" event={"ID":"54adce1e-7ecb-40cb-9d22-8ee7fde39dc1","Type":"ContainerDied","Data":"dbdbe77a50fd47ac9d56df7e4578da1620e899a511dd1c4586dd64633025950e"} Oct 30 22:25:47.645480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.645377 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54" Oct 30 22:25:47.656509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.656492 4991 scope.go:122] "RemoveContainer" containerID="645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c" Oct 30 22:25:47.656810 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:25:47.656792 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c\": container with ID starting with 645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c not found: ID does not exist" containerID="645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c" Oct 30 22:25:47.656873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.656821 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c"} err="failed to get container status \"645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c\": rpc error: code = NotFound desc = could not find container \"645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c\": container with ID starting with 645bfdfbe5d2bbba89357777cacbadc581efdf2d1545518bf06c8dfa25a9700c not found: ID does not exist" Oct 30 22:25:47.656873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.656842 4991 scope.go:122] "RemoveContainer" containerID="81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526" Oct 30 22:25:47.668459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.668439 4991 scope.go:122] "RemoveContainer" containerID="81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526" Oct 30 22:25:47.668720 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:25:47.668699 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526\": container with ID starting with 81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526 not found: ID does not exist" containerID="81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526" Oct 30 22:25:47.668789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.668728 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526"} err="failed to get container status \"81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526\": rpc error: code = NotFound desc = could not find container \"81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526\": container with ID starting with 81df813c23a10b45f8e55c50490237aecb871828460cb624c55164804ecb5526 not found: ID does not exist" Oct 30 22:25:47.802160 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.802124 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j"] Oct 30 22:25:47.806834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.806807 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bd7fe-64bddb984f-bf47j"] Oct 30 22:25:47.876241 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.876214 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de363edf-a0ea-4b46-b80c-53157e82dbf5" path="/var/lib/kubelet/pods/de363edf-a0ea-4b46-b80c-53157e82dbf5/volumes" Oct 30 22:25:47.911493 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:25:47.911406 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:25:47.911648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.911510 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54"] Oct 30 22:25:47.914889 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:47.914868 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-46w54"] Oct 30 22:25:49.874827 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:49.874795 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" path="/var/lib/kubelet/pods/54adce1e-7ecb-40cb-9d22-8ee7fde39dc1/volumes" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421191 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr"] Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421884 4991 cpu_manager.go:414] "containerMap: removing container" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421902 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421917 4991 cpu_manager.go:414] "containerMap: removing container" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421922 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421936 4991 cpu_manager.go:414] "containerMap: removing container" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421941 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421958 4991 cpu_manager.go:414] "containerMap: removing container" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421966 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421974 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421978 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.421998 4991 cpu_manager.go:414] "containerMap: removing container" podUID="de363edf-a0ea-4b46-b80c-53157e82dbf5" containerName="maas-api" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.422002 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de363edf-a0ea-4b46-b80c-53157e82dbf5" containerName="maas-api" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.422096 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="de363edf-a0ea-4b46-b80c-53157e82dbf5" containerName="maas-api" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.422106 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe4e4a8-11eb-4720-bfac-a62d1136d8bd" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.422118 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ac1480-8a43-404b-9787-ed5d05d9427d" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.422128 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cf0461-d0c9-4bc7-8c21-4572cb9fc95e" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.422134 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="54adce1e-7ecb-40cb-9d22-8ee7fde39dc1" containerName="istio-proxy" Oct 30 22:25:50.428176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.422142 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b2c50f-940b-4d9c-9c27-4f83b7667a78" containerName="istio-proxy" Oct 30 22:25:50.434007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.433973 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.493351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.491682 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cf5019-openshift-default-dockercfg-99ggc\"" Oct 30 22:25:50.501364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501270 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13dea8-9817-452c-a658-ce355e4b0de6-istiod-ca-cert\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501352 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-token\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501412 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4lj\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-kube-api-access-vs4lj\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501470 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-socket\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501509 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-podinfo\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501538 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-credential-socket\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501686 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501639 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-certs\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501686 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501677 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-data\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.501751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.501731 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-envoy\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.543178 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.543144 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr"] Oct 30 22:25:50.602357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602311 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-certs\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602371 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-data\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602397 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-envoy\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602416 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13dea8-9817-452c-a658-ce355e4b0de6-istiod-ca-cert\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602435 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-token\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602501 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4lj\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-kube-api-access-vs4lj\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602547 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-socket\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602574 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-podinfo\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602595 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-credential-socket\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.602974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.602957 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-credential-socket\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.603138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.603122 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-certs\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.603311 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.603296 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-data\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.603917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.603887 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-socket\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.604221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.604199 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13dea8-9817-452c-a658-ce355e4b0de6-istiod-ca-cert\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.605863 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.605840 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-envoy\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.606567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.606546 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-podinfo\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.662028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.661996 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4lj\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-kube-api-access-vs4lj\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.663700 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.663675 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-token\") pod \"e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:50.760875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:50.760832 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:51.194566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.194488 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr"] Oct 30 22:25:51.200480 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:25:51.200449 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b13dea8_9817_452c_a658_ce355e4b0de6.slice/crio-fb689fe2d7a56dbca6f7d14b6e2a52411f5df60ff196860aecb23fc411e570dc WatchSource:0}: Error finding container fb689fe2d7a56dbca6f7d14b6e2a52411f5df60ff196860aecb23fc411e570dc: Status 404 returned error can't find the container with id fb689fe2d7a56dbca6f7d14b6e2a52411f5df60ff196860aecb23fc411e570dc Oct 30 22:25:51.203480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.203451 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:25:51.203555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.203517 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:25:51.203555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.203547 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:25:51.676538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.676492 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" event={"ID":"0b13dea8-9817-452c-a658-ce355e4b0de6","Type":"ContainerStarted","Data":"0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b"} Oct 30 22:25:51.676706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.676558 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" event={"ID":"0b13dea8-9817-452c-a658-ce355e4b0de6","Type":"ContainerStarted","Data":"fb689fe2d7a56dbca6f7d14b6e2a52411f5df60ff196860aecb23fc411e570dc"} Oct 30 22:25:51.761761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.761732 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:51.836863 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:51.836812 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" podStartSLOduration=1.836797098 podStartE2EDuration="1.836797098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:51.836211631 +0000 UTC m=+2372.135621513" watchObservedRunningTime="2026-10-30 22:25:51.836797098 +0000 UTC m=+2372.136206976" Oct 30 22:25:52.767689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:52.767660 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:53.690659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:53.690632 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:53.691529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:53.691501 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:25:56.555880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:56.555848 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75"] Oct 30 22:25:56.556453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:25:56.556071 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" containerID="cri-o://afdf02e98516598e492dcfc578ec9c86bfe6247fa2dfd378bff96856f685a40a" gracePeriod=30 Oct 30 22:26:01.772951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.772909 4991 generic.go:356] "Generic (PLEG): container finished" podID="1905bbf4-da83-45f1-addc-018a5e968d76" containerID="afdf02e98516598e492dcfc578ec9c86bfe6247fa2dfd378bff96856f685a40a" exitCode=0 Oct 30 22:26:01.773450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.773028 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" event={"ID":"1905bbf4-da83-45f1-addc-018a5e968d76","Type":"ContainerDied","Data":"afdf02e98516598e492dcfc578ec9c86bfe6247fa2dfd378bff96856f685a40a"} Oct 30 22:26:01.773450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.773127 4991 scope.go:122] "RemoveContainer" containerID="16b7cb7a97ae1da1932d6e3d967bdc091f8143dfdef00a4a95c75bce6401e84f" Oct 30 22:26:01.821445 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.821389 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:26:01.948153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948121 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-kube-api-access-gwkzg\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-kube-api-access-gwkzg\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948171 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1905bbf4-da83-45f1-addc-018a5e968d76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1905bbf4-da83-45f1-addc-018a5e968d76-istio-podinfo\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948202 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-credential-socket\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948251 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-istio-token\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-istio-token\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948307 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-socket\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948340 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-certs\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948653 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948363 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-envoy\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948653 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948425 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-data\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948653 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948454 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1905bbf4-da83-45f1-addc-018a5e968d76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1905bbf4-da83-45f1-addc-018a5e968d76-istiod-ca-cert\") pod \"1905bbf4-da83-45f1-addc-018a5e968d76\" (UID: \"1905bbf4-da83-45f1-addc-018a5e968d76\") " Oct 30 22:26:01.948653 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948575 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-credential-socket" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.948872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948657 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-socket" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.948872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.948844 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-certs" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.949043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.949020 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1905bbf4-da83-45f1-addc-018a5e968d76-istiod-ca-cert" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:01.949043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.949025 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.949171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.949052 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-data" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.949171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.949059 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.949171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.949091 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.951028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.951000 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-istio-token" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:01.951153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.951071 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1905bbf4-da83-45f1-addc-018a5e968d76-istio-podinfo" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:01.951266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.951246 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-kube-api-access-gwkzg" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "kube-api-access-gwkzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:01.951362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:01.951282 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-envoy" pod "1905bbf4-da83-45f1-addc-018a5e968d76" (UID: "1905bbf4-da83-45f1-addc-018a5e968d76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:02.050995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.050954 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.050995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.050991 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.051191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.051007 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1905bbf4-da83-45f1-addc-018a5e968d76-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.051191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.051021 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1905bbf4-da83-45f1-addc-018a5e968d76-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.051191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.051038 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwkzg\" (UniqueName: \"kubernetes.io/projected/1905bbf4-da83-45f1-addc-018a5e968d76-kube-api-access-gwkzg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.051191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.051053 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1905bbf4-da83-45f1-addc-018a5e968d76-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.782536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.782507 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" Oct 30 22:26:02.782536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.782516 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75" event={"ID":"1905bbf4-da83-45f1-addc-018a5e968d76","Type":"ContainerDied","Data":"c9ab38f601c2511d8d14d2705cc9bf13e0efd8ab1f7c0ee71b7c2847c6c19fe8"} Oct 30 22:26:02.783034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.782586 4991 scope.go:122] "RemoveContainer" containerID="afdf02e98516598e492dcfc578ec9c86bfe6247fa2dfd378bff96856f685a40a" Oct 30 22:26:02.989099 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:02.989055 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75"] Oct 30 22:26:03.034955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:03.034881 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bd7fe-openshift-default-64596c8d97-jgh75"] Oct 30 22:26:03.877267 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:03.877231 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" path="/var/lib/kubelet/pods/1905bbf4-da83-45f1-addc-018a5e968d76/volumes" Oct 30 22:26:13.712566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:13.712533 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz"] Oct 30 22:26:13.713029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:13.712762 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" podUID="fd453ded-f246-4c99-aca7-a2a4d3434eb3" containerName="payload-processing" containerID="cri-o://4de61207d7ab0a78fbdd338aeb9f469fb23d4c067fb423946cd48c8379d2de13" gracePeriod=30 Oct 30 22:26:13.814009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:13.813972 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv"] Oct 30 22:26:13.814296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:13.814246 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" podUID="87c88f73-6294-4180-a08a-859f1090acf8" containerName="payload-pre-processing" containerID="cri-o://082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443" gracePeriod=30 Oct 30 22:26:13.856908 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:13.856878 4991 generic.go:356] "Generic (PLEG): container finished" podID="fd453ded-f246-4c99-aca7-a2a4d3434eb3" containerID="4de61207d7ab0a78fbdd338aeb9f469fb23d4c067fb423946cd48c8379d2de13" exitCode=0 Oct 30 22:26:13.857038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:13.856948 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" event={"ID":"fd453ded-f246-4c99-aca7-a2a4d3434eb3","Type":"ContainerDied","Data":"4de61207d7ab0a78fbdd338aeb9f469fb23d4c067fb423946cd48c8379d2de13"} Oct 30 22:26:13.995874 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:13.995846 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:26:14.102134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.102109 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd453ded-f246-4c99-aca7-a2a4d3434eb3-kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/fd453ded-f246-4c99-aca7-a2a4d3434eb3-kube-api-access-xstl7\") pod \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " Oct 30 22:26:14.102258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.102183 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd453ded-f246-4c99-aca7-a2a4d3434eb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd453ded-f246-4c99-aca7-a2a4d3434eb3-plugins-config-volume\") pod \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\" (UID: \"fd453ded-f246-4c99-aca7-a2a4d3434eb3\") " Oct 30 22:26:14.102597 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.102573 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd453ded-f246-4c99-aca7-a2a4d3434eb3-plugins-config-volume" pod "fd453ded-f246-4c99-aca7-a2a4d3434eb3" (UID: "fd453ded-f246-4c99-aca7-a2a4d3434eb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:14.104366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.104293 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd453ded-f246-4c99-aca7-a2a4d3434eb3-kube-api-access-xstl7" pod "fd453ded-f246-4c99-aca7-a2a4d3434eb3" (UID: "fd453ded-f246-4c99-aca7-a2a4d3434eb3"). InnerVolumeSpecName "kube-api-access-xstl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:14.118083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.118050 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:26:14.203289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.203240 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/fd453ded-f246-4c99-aca7-a2a4d3434eb3-kube-api-access-xstl7\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:14.203289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.203280 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd453ded-f246-4c99-aca7-a2a4d3434eb3-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:14.303772 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.303736 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c88f73-6294-4180-a08a-859f1090acf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c88f73-6294-4180-a08a-859f1090acf8-plugins-config-volume\") pod \"87c88f73-6294-4180-a08a-859f1090acf8\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " Oct 30 22:26:14.303966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.303837 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c88f73-6294-4180-a08a-859f1090acf8-kube-api-access-b9r2g\" (UniqueName: \"kubernetes.io/projected/87c88f73-6294-4180-a08a-859f1090acf8-kube-api-access-b9r2g\") pod \"87c88f73-6294-4180-a08a-859f1090acf8\" (UID: \"87c88f73-6294-4180-a08a-859f1090acf8\") " Oct 30 22:26:14.304178 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.304141 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c88f73-6294-4180-a08a-859f1090acf8-plugins-config-volume" pod "87c88f73-6294-4180-a08a-859f1090acf8" (UID: "87c88f73-6294-4180-a08a-859f1090acf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:14.306011 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.305987 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c88f73-6294-4180-a08a-859f1090acf8-kube-api-access-b9r2g" pod "87c88f73-6294-4180-a08a-859f1090acf8" (UID: "87c88f73-6294-4180-a08a-859f1090acf8"). InnerVolumeSpecName "kube-api-access-b9r2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:14.405565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.405514 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c88f73-6294-4180-a08a-859f1090acf8-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:14.405565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.405568 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9r2g\" (UniqueName: \"kubernetes.io/projected/87c88f73-6294-4180-a08a-859f1090acf8-kube-api-access-b9r2g\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:14.865291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.865250 4991 generic.go:356] "Generic (PLEG): container finished" podID="87c88f73-6294-4180-a08a-859f1090acf8" containerID="082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443" exitCode=0 Oct 30 22:26:14.865787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.865298 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" Oct 30 22:26:14.865787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.865360 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" event={"ID":"87c88f73-6294-4180-a08a-859f1090acf8","Type":"ContainerDied","Data":"082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443"} Oct 30 22:26:14.865787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.865415 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv" event={"ID":"87c88f73-6294-4180-a08a-859f1090acf8","Type":"ContainerDied","Data":"57d05a578f61b2028a8ebfa99c8ea30634423efbda1aef9512dbd53153518b44"} Oct 30 22:26:14.865787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.865447 4991 scope.go:122] "RemoveContainer" containerID="082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443" Oct 30 22:26:14.867726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.867161 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" event={"ID":"fd453ded-f246-4c99-aca7-a2a4d3434eb3","Type":"ContainerDied","Data":"6a9be8ab8b92fda87eb0499ab2f9902ea281339471bbf4759c558089bf5e23af"} Oct 30 22:26:14.867726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.867218 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz" Oct 30 22:26:14.891775 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.891746 4991 scope.go:122] "RemoveContainer" containerID="082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443" Oct 30 22:26:14.892929 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:14.892895 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443\": container with ID starting with 082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443 not found: ID does not exist" containerID="082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443" Oct 30 22:26:14.893056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.892940 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443"} err="failed to get container status \"082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443\": rpc error: code = NotFound desc = could not find container \"082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443\": container with ID starting with 082e87e20cd3b45171a4e5c7f22f691b7bde34a226a23e30ae10355df2d4f443 not found: ID does not exist" Oct 30 22:26:14.893056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:14.892965 4991 scope.go:122] "RemoveContainer" containerID="4de61207d7ab0a78fbdd338aeb9f469fb23d4c067fb423946cd48c8379d2de13" Oct 30 22:26:15.022338 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:15.022243 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv"] Oct 30 22:26:15.068665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:15.068631 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-fd5fbf895-frbjv"] Oct 30 22:26:15.129075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:15.129038 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz"] Oct 30 22:26:15.135907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:15.135876 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-75596d59bd-g5dwz"] Oct 30 22:26:15.883061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:15.883025 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c88f73-6294-4180-a08a-859f1090acf8" path="/var/lib/kubelet/pods/87c88f73-6294-4180-a08a-859f1090acf8/volumes" Oct 30 22:26:15.883596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:15.883501 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd453ded-f246-4c99-aca7-a2a4d3434eb3" path="/var/lib/kubelet/pods/fd453ded-f246-4c99-aca7-a2a4d3434eb3/volumes" Oct 30 22:26:17.911487 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:17.911453 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:26:20.288856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:20.288826 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:26:20.305419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:20.305395 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:26:35.980615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:35.980575 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm"] Oct 30 22:26:35.981181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:35.980834 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" podUID="7b198550-6a6d-42d1-b34b-251b9921ea49" containerName="payload-processing" containerID="cri-o://beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830" gracePeriod=30 Oct 30 22:26:36.031920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.031883 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf"] Oct 30 22:26:36.032098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.032077 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" podUID="a9e6dafd-7a9b-4246-bcb4-b64cc711332c" containerName="payload-pre-processing" containerID="cri-o://dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7" gracePeriod=30 Oct 30 22:26:36.315902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.315875 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:26:36.359508 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.359484 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:26:36.451388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.450330 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b198550-6a6d-42d1-b34b-251b9921ea49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b198550-6a6d-42d1-b34b-251b9921ea49-plugins-config-volume\") pod \"7b198550-6a6d-42d1-b34b-251b9921ea49\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " Oct 30 22:26:36.451388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.450446 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-plugins-config-volume\") pod \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " Oct 30 22:26:36.451388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.450550 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b198550-6a6d-42d1-b34b-251b9921ea49-kube-api-access-n562l\" (UniqueName: \"kubernetes.io/projected/7b198550-6a6d-42d1-b34b-251b9921ea49-kube-api-access-n562l\") pod \"7b198550-6a6d-42d1-b34b-251b9921ea49\" (UID: \"7b198550-6a6d-42d1-b34b-251b9921ea49\") " Oct 30 22:26:36.451388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.450608 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-kube-api-access-xxzct\" (UniqueName: \"kubernetes.io/projected/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-kube-api-access-xxzct\") pod \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\" (UID: \"a9e6dafd-7a9b-4246-bcb4-b64cc711332c\") " Oct 30 22:26:36.451388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.450716 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b198550-6a6d-42d1-b34b-251b9921ea49-plugins-config-volume" pod "7b198550-6a6d-42d1-b34b-251b9921ea49" (UID: "7b198550-6a6d-42d1-b34b-251b9921ea49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:36.451388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.451069 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b198550-6a6d-42d1-b34b-251b9921ea49-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.451388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.451091 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-plugins-config-volume" pod "a9e6dafd-7a9b-4246-bcb4-b64cc711332c" (UID: "a9e6dafd-7a9b-4246-bcb4-b64cc711332c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:36.453700 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.453669 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-kube-api-access-xxzct" pod "a9e6dafd-7a9b-4246-bcb4-b64cc711332c" (UID: "a9e6dafd-7a9b-4246-bcb4-b64cc711332c"). InnerVolumeSpecName "kube-api-access-xxzct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:36.453882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.453865 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b198550-6a6d-42d1-b34b-251b9921ea49-kube-api-access-n562l" pod "7b198550-6a6d-42d1-b34b-251b9921ea49" (UID: "7b198550-6a6d-42d1-b34b-251b9921ea49"). InnerVolumeSpecName "kube-api-access-n562l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:36.552291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.552197 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.552291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.552231 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n562l\" (UniqueName: \"kubernetes.io/projected/7b198550-6a6d-42d1-b34b-251b9921ea49-kube-api-access-n562l\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.552291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:36.552245 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzct\" (UniqueName: \"kubernetes.io/projected/a9e6dafd-7a9b-4246-bcb4-b64cc711332c-kube-api-access-xxzct\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:37.032490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.032457 4991 generic.go:356] "Generic (PLEG): container finished" podID="a9e6dafd-7a9b-4246-bcb4-b64cc711332c" containerID="dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7" exitCode=0 Oct 30 22:26:37.032940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.032519 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" Oct 30 22:26:37.032940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.032545 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" event={"ID":"a9e6dafd-7a9b-4246-bcb4-b64cc711332c","Type":"ContainerDied","Data":"dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7"} Oct 30 22:26:37.032940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.032612 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf" event={"ID":"a9e6dafd-7a9b-4246-bcb4-b64cc711332c","Type":"ContainerDied","Data":"b202d3d77aa505e32337dbadaaab159fa68e862635e6471de7af7021153f61aa"} Oct 30 22:26:37.032940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.032643 4991 scope.go:122] "RemoveContainer" containerID="dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7" Oct 30 22:26:37.034103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.034048 4991 generic.go:356] "Generic (PLEG): container finished" podID="7b198550-6a6d-42d1-b34b-251b9921ea49" containerID="beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830" exitCode=0 Oct 30 22:26:37.034103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.034099 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" Oct 30 22:26:37.034225 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.034126 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" event={"ID":"7b198550-6a6d-42d1-b34b-251b9921ea49","Type":"ContainerDied","Data":"beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830"} Oct 30 22:26:37.034225 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.034188 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm" event={"ID":"7b198550-6a6d-42d1-b34b-251b9921ea49","Type":"ContainerDied","Data":"12a822e6277477d1aa600fe1d669b8bc8964219e47f2a2621a637164e7add48d"} Oct 30 22:26:37.049205 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.049188 4991 scope.go:122] "RemoveContainer" containerID="dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7" Oct 30 22:26:37.049528 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:37.049512 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7\": container with ID starting with dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7 not found: ID does not exist" containerID="dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7" Oct 30 22:26:37.049588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.049537 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7"} err="failed to get container status \"dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7\": rpc error: code = NotFound desc = could not find container \"dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7\": container with ID starting with dab373e72a5931bc9ea718acb6ac079dac82c95778dc389a8b365fed133d63f7 not found: ID does not exist" Oct 30 22:26:37.049588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.049552 4991 scope.go:122] "RemoveContainer" containerID="beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830" Oct 30 22:26:37.061214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.061196 4991 scope.go:122] "RemoveContainer" containerID="beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830" Oct 30 22:26:37.061536 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:37.061517 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830\": container with ID starting with beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830 not found: ID does not exist" containerID="beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830" Oct 30 22:26:37.061621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.061547 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830"} err="failed to get container status \"beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830\": rpc error: code = NotFound desc = could not find container \"beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830\": container with ID starting with beff28f71ee9cdb859ab1b3895ca4d7a9736e196cdf9248a0977c9a8c35d4830 not found: ID does not exist" Oct 30 22:26:37.193361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.193307 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm"] Oct 30 22:26:37.240001 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.239971 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-594bc8cc8-hhfqm"] Oct 30 22:26:37.300461 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.300390 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf"] Oct 30 22:26:37.305252 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.305228 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-5956df65bc-9z8zf"] Oct 30 22:26:37.875075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.875039 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b198550-6a6d-42d1-b34b-251b9921ea49" path="/var/lib/kubelet/pods/7b198550-6a6d-42d1-b34b-251b9921ea49/volumes" Oct 30 22:26:37.875501 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:37.875484 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e6dafd-7a9b-4246-bcb4-b64cc711332c" path="/var/lib/kubelet/pods/a9e6dafd-7a9b-4246-bcb4-b64cc711332c/volumes" Oct 30 22:26:47.911531 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:47.911495 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:26:51.711983 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.711946 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx"] Oct 30 22:26:51.712486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712471 4991 cpu_manager.go:414] "containerMap: removing container" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" Oct 30 22:26:51.712535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712489 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" Oct 30 22:26:51.712535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712505 4991 cpu_manager.go:414] "containerMap: removing container" podUID="fd453ded-f246-4c99-aca7-a2a4d3434eb3" containerName="payload-processing" Oct 30 22:26:51.712535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712511 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd453ded-f246-4c99-aca7-a2a4d3434eb3" containerName="payload-processing" Oct 30 22:26:51.712535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712521 4991 cpu_manager.go:414] "containerMap: removing container" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" Oct 30 22:26:51.712535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712526 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" Oct 30 22:26:51.712535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712536 4991 cpu_manager.go:414] "containerMap: removing container" podUID="a9e6dafd-7a9b-4246-bcb4-b64cc711332c" containerName="payload-pre-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712542 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e6dafd-7a9b-4246-bcb4-b64cc711332c" containerName="payload-pre-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712551 4991 cpu_manager.go:414] "containerMap: removing container" podUID="87c88f73-6294-4180-a08a-859f1090acf8" containerName="payload-pre-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712556 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c88f73-6294-4180-a08a-859f1090acf8" containerName="payload-pre-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712574 4991 cpu_manager.go:414] "containerMap: removing container" podUID="7b198550-6a6d-42d1-b34b-251b9921ea49" containerName="payload-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712578 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b198550-6a6d-42d1-b34b-251b9921ea49" containerName="payload-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712666 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd453ded-f246-4c99-aca7-a2a4d3434eb3" containerName="payload-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712675 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712682 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e6dafd-7a9b-4246-bcb4-b64cc711332c" containerName="payload-pre-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712689 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c88f73-6294-4180-a08a-859f1090acf8" containerName="payload-pre-processing" Oct 30 22:26:51.712716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.712698 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b198550-6a6d-42d1-b34b-251b9921ea49" containerName="payload-processing" Oct 30 22:26:51.716244 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.716221 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.814314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.814277 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cf5019-serving-cert\"" Oct 30 22:26:51.814681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.814655 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.814783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.814706 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4nnk\" (UniqueName: \"kubernetes.io/projected/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-kube-api-access-p4nnk\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.814906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.814883 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.815039 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.815017 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-metrics-certs\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.817579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.817558 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx"] Oct 30 22:26:51.915845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.915809 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.915993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.915891 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-metrics-certs\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.915993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.915958 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.916070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.916000 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4nnk\" (UniqueName: \"kubernetes.io/projected/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-kube-api-access-p4nnk\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.916114 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:51.916089 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-serving-cert: secret "maas-api-e2e-praxis-b-cf5019-serving-cert" not found Oct 30 22:26:51.916197 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:51.916162 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls podName:e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:52.416143178 +0000 UTC m=+2432.715553053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls") pod "maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" (UID: "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63") : secret "maas-api-e2e-praxis-b-cf5019-serving-cert" not found Oct 30 22:26:51.916565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.916538 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.918195 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.918176 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-metrics-certs\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:51.979092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:51.979055 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4nnk\" (UniqueName: \"kubernetes.io/projected/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-kube-api-access-p4nnk\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:52.423381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:52.423290 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:52.425793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:52.425760 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:52.630599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:52.630562 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:53.014971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:53.014941 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx"] Oct 30 22:26:53.021850 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:26:53.021818 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38c7c9a_d6bf_4ccc_90e6_3a5cff33ef63.slice/crio-8f22ea1e83c48269dec3567de5308b5d4385ef0ad17ffcab5400839ed8230e66 WatchSource:0}: Error finding container 8f22ea1e83c48269dec3567de5308b5d4385ef0ad17ffcab5400839ed8230e66: Status 404 returned error can't find the container with id 8f22ea1e83c48269dec3567de5308b5d4385ef0ad17ffcab5400839ed8230e66 Oct 30 22:26:53.023572 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:53.023550 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:26:53.153696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:53.153662 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" event={"ID":"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63","Type":"ContainerStarted","Data":"8f22ea1e83c48269dec3567de5308b5d4385ef0ad17ffcab5400839ed8230e66"} Oct 30 22:26:55.173395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:55.173355 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" event={"ID":"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63","Type":"ContainerStarted","Data":"dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab"} Oct 30 22:26:55.173859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:55.173447 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:26:55.287963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:55.287904 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" podStartSLOduration=3.185453276 podStartE2EDuration="4.28789076s" totalImagesPullingTime="1.102437484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:55.284659398 +0000 UTC m=+2435.584069301" watchObservedRunningTime="2026-10-30 22:26:55.28789076 +0000 UTC m=+2435.587300639" Oct 30 22:26:56.455299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.455262 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv"] Oct 30 22:26:56.455736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.455518 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" podUID="8397b25f-e3c5-420e-bcd8-e4c1181a49dc" containerName="maas-api" containerID="cri-o://0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113" gracePeriod=30 Oct 30 22:26:56.744027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.743999 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:26:56.792727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.792692 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-maas-api-tls\") pod \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " Oct 30 22:26:56.792878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.792825 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-metrics-certs\") pod \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " Oct 30 22:26:56.792878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.792858 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-kube-api-access-shxfl\" (UniqueName: \"kubernetes.io/projected/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-kube-api-access-shxfl\") pod \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " Oct 30 22:26:56.793000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.792915 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-openshift-service-ca\") pod \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\" (UID: \"8397b25f-e3c5-420e-bcd8-e4c1181a49dc\") " Oct 30 22:26:56.793334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.793293 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-openshift-service-ca" pod "8397b25f-e3c5-420e-bcd8-e4c1181a49dc" (UID: "8397b25f-e3c5-420e-bcd8-e4c1181a49dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:56.794929 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.794905 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-maas-api-tls" pod "8397b25f-e3c5-420e-bcd8-e4c1181a49dc" (UID: "8397b25f-e3c5-420e-bcd8-e4c1181a49dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:56.795016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.794924 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-metrics-certs" pod "8397b25f-e3c5-420e-bcd8-e4c1181a49dc" (UID: "8397b25f-e3c5-420e-bcd8-e4c1181a49dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:56.795016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.794935 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-kube-api-access-shxfl" pod "8397b25f-e3c5-420e-bcd8-e4c1181a49dc" (UID: "8397b25f-e3c5-420e-bcd8-e4c1181a49dc"). InnerVolumeSpecName "kube-api-access-shxfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:56.893999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.893968 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.893999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.893995 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.893999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.894006 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxfl\" (UniqueName: \"kubernetes.io/projected/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-kube-api-access-shxfl\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.894261 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:56.894015 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8397b25f-e3c5-420e-bcd8-e4c1181a49dc-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:57.186275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.186244 4991 generic.go:356] "Generic (PLEG): container finished" podID="8397b25f-e3c5-420e-bcd8-e4c1181a49dc" containerID="0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113" exitCode=0 Oct 30 22:26:57.186471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.186287 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" Oct 30 22:26:57.186471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.186338 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" event={"ID":"8397b25f-e3c5-420e-bcd8-e4c1181a49dc","Type":"ContainerDied","Data":"0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113"} Oct 30 22:26:57.186471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.186398 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv" event={"ID":"8397b25f-e3c5-420e-bcd8-e4c1181a49dc","Type":"ContainerDied","Data":"6e50db6b35a52200a6a4256cb134d83068e9ef35c63c73c6db92c56ef6a35676"} Oct 30 22:26:57.186471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.186420 4991 scope.go:122] "RemoveContainer" containerID="0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113" Oct 30 22:26:57.196980 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.196960 4991 scope.go:122] "RemoveContainer" containerID="0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113" Oct 30 22:26:57.197233 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:26:57.197216 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113\": container with ID starting with 0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113 not found: ID does not exist" containerID="0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113" Oct 30 22:26:57.197280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.197242 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113"} err="failed to get container status \"0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113\": rpc error: code = NotFound desc = could not find container \"0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113\": container with ID starting with 0e8c80b25dd91531b5b869989981bca67a4db0140cd53e17bde07eb6ed48c113 not found: ID does not exist" Oct 30 22:26:57.342420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.342379 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv"] Oct 30 22:26:57.349132 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.349105 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c89e47-675455947c-kj8sv"] Oct 30 22:26:57.876297 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:26:57.876262 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8397b25f-e3c5-420e-bcd8-e4c1181a49dc" path="/var/lib/kubelet/pods/8397b25f-e3c5-420e-bcd8-e4c1181a49dc/volumes" Oct 30 22:27:01.186416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:01.186387 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:27:02.596090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.596054 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj"] Oct 30 22:27:02.596583 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.596311 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" podUID="4efc4ee7-85a6-4338-aad3-5735bbea322e" containerName="maas-api" containerID="cri-o://6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48" gracePeriod=30 Oct 30 22:27:02.876917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.876887 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:27:02.962114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.961952 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efc4ee7-85a6-4338-aad3-5735bbea322e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efc4ee7-85a6-4338-aad3-5735bbea322e-openshift-service-ca\") pod \"4efc4ee7-85a6-4338-aad3-5735bbea322e\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " Oct 30 22:27:02.962114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.962015 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efc4ee7-85a6-4338-aad3-5735bbea322e-kube-api-access-ltbtw\" (UniqueName: \"kubernetes.io/projected/4efc4ee7-85a6-4338-aad3-5735bbea322e-kube-api-access-ltbtw\") pod \"4efc4ee7-85a6-4338-aad3-5735bbea322e\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " Oct 30 22:27:02.963526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.962309 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls\") pod \"4efc4ee7-85a6-4338-aad3-5735bbea322e\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " Oct 30 22:27:02.963526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.962474 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efc4ee7-85a6-4338-aad3-5735bbea322e-openshift-service-ca" pod "4efc4ee7-85a6-4338-aad3-5735bbea322e" (UID: "4efc4ee7-85a6-4338-aad3-5735bbea322e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:02.963526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.962555 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-metrics-certs\") pod \"4efc4ee7-85a6-4338-aad3-5735bbea322e\" (UID: \"4efc4ee7-85a6-4338-aad3-5735bbea322e\") " Oct 30 22:27:02.963526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.962951 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efc4ee7-85a6-4338-aad3-5735bbea322e-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:02.977349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.975845 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-metrics-certs" pod "4efc4ee7-85a6-4338-aad3-5735bbea322e" (UID: "4efc4ee7-85a6-4338-aad3-5735bbea322e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:02.984380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.981486 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls" pod "4efc4ee7-85a6-4338-aad3-5735bbea322e" (UID: "4efc4ee7-85a6-4338-aad3-5735bbea322e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:02.984380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:02.981594 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efc4ee7-85a6-4338-aad3-5735bbea322e-kube-api-access-ltbtw" pod "4efc4ee7-85a6-4338-aad3-5735bbea322e" (UID: "4efc4ee7-85a6-4338-aad3-5735bbea322e"). InnerVolumeSpecName "kube-api-access-ltbtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:03.064314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.064275 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:03.064314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.064315 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efc4ee7-85a6-4338-aad3-5735bbea322e-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:03.064596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.064367 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbtw\" (UniqueName: \"kubernetes.io/projected/4efc4ee7-85a6-4338-aad3-5735bbea322e-kube-api-access-ltbtw\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:03.229373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.229308 4991 generic.go:356] "Generic (PLEG): container finished" podID="4efc4ee7-85a6-4338-aad3-5735bbea322e" containerID="6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48" exitCode=0 Oct 30 22:27:03.229373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.229353 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" event={"ID":"4efc4ee7-85a6-4338-aad3-5735bbea322e","Type":"ContainerDied","Data":"6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48"} Oct 30 22:27:03.229632 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.229377 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" Oct 30 22:27:03.229632 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.229426 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj" event={"ID":"4efc4ee7-85a6-4338-aad3-5735bbea322e","Type":"ContainerDied","Data":"85bc4027f0204f689b123f048914ba1252a47ef624e3d84d588f2a67bdcf143d"} Oct 30 22:27:03.229632 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.229459 4991 scope.go:122] "RemoveContainer" containerID="6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48" Oct 30 22:27:03.241784 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.241765 4991 scope.go:122] "RemoveContainer" containerID="6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48" Oct 30 22:27:03.242101 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:27:03.242080 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48\": container with ID starting with 6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48 not found: ID does not exist" containerID="6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48" Oct 30 22:27:03.242168 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.242110 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48"} err="failed to get container status \"6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48\": rpc error: code = NotFound desc = could not find container \"6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48\": container with ID starting with 6765fcf3f7d30e7aaef4c19253f6323d82b813bc9f24990717db224cceb7ae48 not found: ID does not exist" Oct 30 22:27:03.387179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.387143 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj"] Oct 30 22:27:03.435389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.435351 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebcc6d-55f7cc94dd-g2qdj"] Oct 30 22:27:03.874755 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:03.874717 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efc4ee7-85a6-4338-aad3-5735bbea322e" path="/var/lib/kubelet/pods/4efc4ee7-85a6-4338-aad3-5735bbea322e/volumes" Oct 30 22:27:08.481845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.481804 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g"] Oct 30 22:27:08.482923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.482896 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8397b25f-e3c5-420e-bcd8-e4c1181a49dc" containerName="maas-api" Oct 30 22:27:08.482923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.482920 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8397b25f-e3c5-420e-bcd8-e4c1181a49dc" containerName="maas-api" Oct 30 22:27:08.483125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.482999 4991 cpu_manager.go:414] "containerMap: removing container" podUID="4efc4ee7-85a6-4338-aad3-5735bbea322e" containerName="maas-api" Oct 30 22:27:08.483125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.483008 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efc4ee7-85a6-4338-aad3-5735bbea322e" containerName="maas-api" Oct 30 22:27:08.483125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.483122 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efc4ee7-85a6-4338-aad3-5735bbea322e" containerName="maas-api" Oct 30 22:27:08.483310 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.483137 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8397b25f-e3c5-420e-bcd8-e4c1181a49dc" containerName="maas-api" Oct 30 22:27:08.483310 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.483157 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="1905bbf4-da83-45f1-addc-018a5e968d76" containerName="istio-proxy" Oct 30 22:27:08.489116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.489075 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.539510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.539476 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-459aa8-openshift-default-dockercfg-n7vs9\"" Oct 30 22:27:08.588673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.588644 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g"] Oct 30 22:27:08.624788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.624753 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-socket\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.624788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.624790 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-data\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.624986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.624830 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-envoy\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.624986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.624865 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-token\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.624986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.624932 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d199c57d-49ec-4b6b-8cee-6511796ee6de-istiod-ca-cert\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.625089 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.625066 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5f6m\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-kube-api-access-v5f6m\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.625127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.625102 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-certs\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.625202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.625142 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-podinfo\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.625282 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.625261 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-credential-socket\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726204 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-socket\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726238 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-data\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726276 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-envoy\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726299 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-token\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726338 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d199c57d-49ec-4b6b-8cee-6511796ee6de-istiod-ca-cert\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726410 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5f6m\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-kube-api-access-v5f6m\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726477 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-certs\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726542 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-podinfo\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726636 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-credential-socket\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726696 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-socket\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.726878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726754 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-data\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.727044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.726940 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-credential-socket\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.727219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.727187 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-certs\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.727386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.727233 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d199c57d-49ec-4b6b-8cee-6511796ee6de-istiod-ca-cert\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.729486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.729466 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-envoy\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.729829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.729808 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-podinfo\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.784569 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.784500 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-token\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.786583 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.786561 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5f6m\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-kube-api-access-v5f6m\") pod \"e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:08.822642 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:08.822618 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:09.265570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:09.265536 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g"] Oct 30 22:27:09.277005 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:09.276963 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:09.277133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:09.277050 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:09.277133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:09.277121 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:10.290836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:10.290791 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" event={"ID":"d199c57d-49ec-4b6b-8cee-6511796ee6de","Type":"ContainerStarted","Data":"a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927"} Oct 30 22:27:10.291223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:10.290853 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" event={"ID":"d199c57d-49ec-4b6b-8cee-6511796ee6de","Type":"ContainerStarted","Data":"78312b5781a18baed0e40ad67d7b24ec8a4490282b158e23f594637d0b454a08"} Oct 30 22:27:10.452312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:10.452263 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" podStartSLOduration=2.452248626 podStartE2EDuration="2.452248626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:10.450810963 +0000 UTC m=+2450.750220901" watchObservedRunningTime="2026-10-30 22:27:10.452248626 +0000 UTC m=+2450.751658505" Oct 30 22:27:10.823511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:10.823472 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:10.829453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:10.829425 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:11.298309 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:11.298280 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:11.299429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:11.299404 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:27:14.969108 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:14.969072 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v"] Oct 30 22:27:14.969523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:14.969255 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" podUID="9eb25b59-8b56-4d2d-bb55-6c9854e1624d" containerName="maas-api" containerID="cri-o://484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe" gracePeriod=30 Oct 30 22:27:15.251152 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.251127 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:27:15.305144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.305112 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls\") pod \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " Oct 30 22:27:15.305307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.305223 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-kube-api-access-8xpzc\" (UniqueName: \"kubernetes.io/projected/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-kube-api-access-8xpzc\") pod \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " Oct 30 22:27:15.305307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.305256 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-openshift-service-ca\") pod \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " Oct 30 22:27:15.305479 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.305395 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-metrics-certs\") pod \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\" (UID: \"9eb25b59-8b56-4d2d-bb55-6c9854e1624d\") " Oct 30 22:27:15.305672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.305647 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-openshift-service-ca" pod "9eb25b59-8b56-4d2d-bb55-6c9854e1624d" (UID: "9eb25b59-8b56-4d2d-bb55-6c9854e1624d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:15.305938 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.305917 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.307584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.307558 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-metrics-certs" pod "9eb25b59-8b56-4d2d-bb55-6c9854e1624d" (UID: "9eb25b59-8b56-4d2d-bb55-6c9854e1624d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:15.307684 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.307637 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-kube-api-access-8xpzc" pod "9eb25b59-8b56-4d2d-bb55-6c9854e1624d" (UID: "9eb25b59-8b56-4d2d-bb55-6c9854e1624d"). InnerVolumeSpecName "kube-api-access-8xpzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:15.308133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.308111 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls" pod "9eb25b59-8b56-4d2d-bb55-6c9854e1624d" (UID: "9eb25b59-8b56-4d2d-bb55-6c9854e1624d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:15.328871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.328847 4991 generic.go:356] "Generic (PLEG): container finished" podID="9eb25b59-8b56-4d2d-bb55-6c9854e1624d" containerID="484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe" exitCode=0 Oct 30 22:27:15.328957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.328888 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" Oct 30 22:27:15.328957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.328931 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" event={"ID":"9eb25b59-8b56-4d2d-bb55-6c9854e1624d","Type":"ContainerDied","Data":"484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe"} Oct 30 22:27:15.329035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.328981 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v" event={"ID":"9eb25b59-8b56-4d2d-bb55-6c9854e1624d","Type":"ContainerDied","Data":"c7c36ba5d2fdd856a8d34e9687eb72f79f584c33af86b526c105ae8ab5a23514"} Oct 30 22:27:15.329035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.329007 4991 scope.go:122] "RemoveContainer" containerID="484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe" Oct 30 22:27:15.344002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.343923 4991 scope.go:122] "RemoveContainer" containerID="484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe" Oct 30 22:27:15.345066 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:27:15.345027 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe\": container with ID starting with 484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe not found: ID does not exist" containerID="484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe" Oct 30 22:27:15.345155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.345066 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe"} err="failed to get container status \"484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe\": rpc error: code = NotFound desc = could not find container \"484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe\": container with ID starting with 484f3af9c996100be0bb1436148b370919efa11ed54deac36e5cd996e8d214fe not found: ID does not exist" Oct 30 22:27:15.417984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.417952 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xpzc\" (UniqueName: \"kubernetes.io/projected/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-kube-api-access-8xpzc\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.417984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.417981 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.418217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.417996 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eb25b59-8b56-4d2d-bb55-6c9854e1624d-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.461079 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.461038 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv"] Oct 30 22:27:15.461236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.461223 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" podUID="b4798bcb-37da-4f81-aad9-0ef7ac6f4372" containerName="payload-processing" containerID="cri-o://30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340" gracePeriod=30 Oct 30 22:27:15.476049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.476021 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v"] Oct 30 22:27:15.481933 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.481908 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a3357-86c48c44b6-xcj7v"] Oct 30 22:27:15.515247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.515225 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp"] Oct 30 22:27:15.515465 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.515429 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" podUID="8fc60c98-0d71-4540-886b-6fe4765d7cff" containerName="payload-pre-processing" containerID="cri-o://d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791" gracePeriod=30 Oct 30 22:27:15.775289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.775251 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:27:15.812537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.812514 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:27:15.875668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.875637 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb25b59-8b56-4d2d-bb55-6c9854e1624d" path="/var/lib/kubelet/pods/9eb25b59-8b56-4d2d-bb55-6c9854e1624d/volumes" Oct 30 22:27:15.924681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.924648 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-plugins-config-volume\") pod \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " Oct 30 22:27:15.924845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.924788 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc60c98-0d71-4540-886b-6fe4765d7cff-kube-api-access-tf6x5\" (UniqueName: \"kubernetes.io/projected/8fc60c98-0d71-4540-886b-6fe4765d7cff-kube-api-access-tf6x5\") pod \"8fc60c98-0d71-4540-886b-6fe4765d7cff\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " Oct 30 22:27:15.924845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.924821 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-kube-api-access-gh428\" (UniqueName: \"kubernetes.io/projected/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-kube-api-access-gh428\") pod \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\" (UID: \"b4798bcb-37da-4f81-aad9-0ef7ac6f4372\") " Oct 30 22:27:15.924963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.924861 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc60c98-0d71-4540-886b-6fe4765d7cff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc60c98-0d71-4540-886b-6fe4765d7cff-plugins-config-volume\") pod \"8fc60c98-0d71-4540-886b-6fe4765d7cff\" (UID: \"8fc60c98-0d71-4540-886b-6fe4765d7cff\") " Oct 30 22:27:15.925309 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.925262 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-plugins-config-volume" pod "b4798bcb-37da-4f81-aad9-0ef7ac6f4372" (UID: "b4798bcb-37da-4f81-aad9-0ef7ac6f4372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:15.925579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.925557 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc60c98-0d71-4540-886b-6fe4765d7cff-plugins-config-volume" pod "8fc60c98-0d71-4540-886b-6fe4765d7cff" (UID: "8fc60c98-0d71-4540-886b-6fe4765d7cff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:15.927675 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.927551 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-kube-api-access-gh428" pod "b4798bcb-37da-4f81-aad9-0ef7ac6f4372" (UID: "b4798bcb-37da-4f81-aad9-0ef7ac6f4372"). InnerVolumeSpecName "kube-api-access-gh428". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:15.927675 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:15.927642 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc60c98-0d71-4540-886b-6fe4765d7cff-kube-api-access-tf6x5" pod "8fc60c98-0d71-4540-886b-6fe4765d7cff" (UID: "8fc60c98-0d71-4540-886b-6fe4765d7cff"). InnerVolumeSpecName "kube-api-access-tf6x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:16.028189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.028146 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:16.028189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.028185 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6x5\" (UniqueName: \"kubernetes.io/projected/8fc60c98-0d71-4540-886b-6fe4765d7cff-kube-api-access-tf6x5\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:16.028753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.028215 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh428\" (UniqueName: \"kubernetes.io/projected/b4798bcb-37da-4f81-aad9-0ef7ac6f4372-kube-api-access-gh428\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:16.028753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.028232 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc60c98-0d71-4540-886b-6fe4765d7cff-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:16.338156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.338071 4991 generic.go:356] "Generic (PLEG): container finished" podID="8fc60c98-0d71-4540-886b-6fe4765d7cff" containerID="d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791" exitCode=0 Oct 30 22:27:16.338156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.338117 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" Oct 30 22:27:16.338433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.338157 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" event={"ID":"8fc60c98-0d71-4540-886b-6fe4765d7cff","Type":"ContainerDied","Data":"d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791"} Oct 30 22:27:16.338433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.338216 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp" event={"ID":"8fc60c98-0d71-4540-886b-6fe4765d7cff","Type":"ContainerDied","Data":"f64e869214385a0773e3e3176f3d41672022b7e4f4e59efe10f390bddf8db79b"} Oct 30 22:27:16.338433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.338244 4991 scope.go:122] "RemoveContainer" containerID="d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791" Oct 30 22:27:16.341359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.339790 4991 generic.go:356] "Generic (PLEG): container finished" podID="b4798bcb-37da-4f81-aad9-0ef7ac6f4372" containerID="30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340" exitCode=0 Oct 30 22:27:16.341359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.339835 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" event={"ID":"b4798bcb-37da-4f81-aad9-0ef7ac6f4372","Type":"ContainerDied","Data":"30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340"} Oct 30 22:27:16.341359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.339876 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" event={"ID":"b4798bcb-37da-4f81-aad9-0ef7ac6f4372","Type":"ContainerDied","Data":"48ed88ef7322f3b2784a36528815f77a6958b79a3a65ca13100d272b01365957"} Oct 30 22:27:16.341359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.339840 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv" Oct 30 22:27:16.352853 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.352827 4991 scope.go:122] "RemoveContainer" containerID="d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791" Oct 30 22:27:16.353129 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:27:16.353109 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791\": container with ID starting with d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791 not found: ID does not exist" containerID="d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791" Oct 30 22:27:16.353180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.353138 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791"} err="failed to get container status \"d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791\": rpc error: code = NotFound desc = could not find container \"d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791\": container with ID starting with d3733e5ae1a0268761b4f2bc9ca84d26ec5210a74e2888e3a0057a2fa99f4791 not found: ID does not exist" Oct 30 22:27:16.353180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.353155 4991 scope.go:122] "RemoveContainer" containerID="30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340" Oct 30 22:27:16.364771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.364748 4991 scope.go:122] "RemoveContainer" containerID="30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340" Oct 30 22:27:16.367658 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:27:16.367626 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340\": container with ID starting with 30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340 not found: ID does not exist" containerID="30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340" Oct 30 22:27:16.367761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.367661 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340"} err="failed to get container status \"30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340\": rpc error: code = NotFound desc = could not find container \"30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340\": container with ID starting with 30ffa0592161cee3187a2d162240d484f93059063f8003f4ccf61a4ace69d340 not found: ID does not exist" Oct 30 22:27:16.497047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.497016 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv"] Oct 30 22:27:16.503813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.503784 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-5685c57f46-2ktgv"] Oct 30 22:27:16.612919 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.612831 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp"] Oct 30 22:27:16.617424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:16.617401 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-5bfd68c7cf-w7tsp"] Oct 30 22:27:17.877446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:17.877406 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc60c98-0d71-4540-886b-6fe4765d7cff" path="/var/lib/kubelet/pods/8fc60c98-0d71-4540-886b-6fe4765d7cff/volumes" Oct 30 22:27:17.877997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:17.877977 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4798bcb-37da-4f81-aad9-0ef7ac6f4372" path="/var/lib/kubelet/pods/b4798bcb-37da-4f81-aad9-0ef7ac6f4372/volumes" Oct 30 22:27:17.911159 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:27:17.911120 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:27:25.263374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:25.263340 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq"] Oct 30 22:27:25.263948 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:25.263559 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" containerID="cri-o://1896233dfe3fbd615365702b549826b6f91a9d2b726259f779166851dd657784" gracePeriod=30 Oct 30 22:27:27.450018 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.449978 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt"] Oct 30 22:27:27.452993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.452952 4991 cpu_manager.go:414] "containerMap: removing container" podUID="b4798bcb-37da-4f81-aad9-0ef7ac6f4372" containerName="payload-processing" Oct 30 22:27:27.453155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.452995 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4798bcb-37da-4f81-aad9-0ef7ac6f4372" containerName="payload-processing" Oct 30 22:27:27.453155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.453036 4991 cpu_manager.go:414] "containerMap: removing container" podUID="9eb25b59-8b56-4d2d-bb55-6c9854e1624d" containerName="maas-api" Oct 30 22:27:27.453155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.453045 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb25b59-8b56-4d2d-bb55-6c9854e1624d" containerName="maas-api" Oct 30 22:27:27.453155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.453084 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8fc60c98-0d71-4540-886b-6fe4765d7cff" containerName="payload-pre-processing" Oct 30 22:27:27.453155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.453106 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc60c98-0d71-4540-886b-6fe4765d7cff" containerName="payload-pre-processing" Oct 30 22:27:27.453459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.453292 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc60c98-0d71-4540-886b-6fe4765d7cff" containerName="payload-pre-processing" Oct 30 22:27:27.453459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.453314 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4798bcb-37da-4f81-aad9-0ef7ac6f4372" containerName="payload-processing" Oct 30 22:27:27.453459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.453366 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb25b59-8b56-4d2d-bb55-6c9854e1624d" containerName="maas-api" Oct 30 22:27:27.466611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.466585 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.516852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.516824 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1875b4-openshift-default-dockercfg-wfpzt\"" Oct 30 22:27:27.570758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.570711 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt"] Oct 30 22:27:27.581624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.581586 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-certs\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.581890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.581864 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e698f07-79f2-451e-8777-e73f923096c1-istio-podinfo\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.582077 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.582058 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-istio-token\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.582302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.582284 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-data\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.584771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.582442 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-socket\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.584968 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.584947 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e698f07-79f2-451e-8777-e73f923096c1-istiod-ca-cert\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.585185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.585144 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-credential-socket\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.585270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.585255 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-envoy\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.585420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.585389 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pfg\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-kube-api-access-57pfg\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.686670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686631 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-data\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.686670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686675 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-socket\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.686917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686729 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e698f07-79f2-451e-8777-e73f923096c1-istiod-ca-cert\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.686917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686756 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-credential-socket\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.686917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686829 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-envoy\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.686917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686896 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pfg\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-kube-api-access-57pfg\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686952 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-certs\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.686981 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e698f07-79f2-451e-8777-e73f923096c1-istio-podinfo\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.687004 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-istio-token\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.687107 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-data\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687413 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.687141 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-socket\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.687437 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-credential-socket\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.687460 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-certs\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.687614 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.687590 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e698f07-79f2-451e-8777-e73f923096c1-istiod-ca-cert\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.689296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.689272 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-envoy\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.689392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.689376 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e698f07-79f2-451e-8777-e73f923096c1-istio-podinfo\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.746798 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.746766 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-istio-token\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.747197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.747173 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pfg\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-kube-api-access-57pfg\") pod \"e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:27.794426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:27.794399 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:28.260960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.260924 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt"] Oct 30 22:27:28.267535 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:27:28.267502 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e698f07_79f2_451e_8777_e73f923096c1.slice/crio-37d528abe7c120b285dd9a129fc2d487c5d8fe695ba43a94c818bd16d7f21aa9 WatchSource:0}: Error finding container 37d528abe7c120b285dd9a129fc2d487c5d8fe695ba43a94c818bd16d7f21aa9: Status 404 returned error can't find the container with id 37d528abe7c120b285dd9a129fc2d487c5d8fe695ba43a94c818bd16d7f21aa9 Oct 30 22:27:28.270070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.270040 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:28.270156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.270111 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:28.270156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.270141 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:28.475238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.475207 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" event={"ID":"9e698f07-79f2-451e-8777-e73f923096c1","Type":"ContainerStarted","Data":"15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422"} Oct 30 22:27:28.475637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.475256 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" event={"ID":"9e698f07-79f2-451e-8777-e73f923096c1","Type":"ContainerStarted","Data":"37d528abe7c120b285dd9a129fc2d487c5d8fe695ba43a94c818bd16d7f21aa9"} Oct 30 22:27:28.594433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.594329 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podStartSLOduration=1.594301487 podStartE2EDuration="1.594301487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:28.592985673 +0000 UTC m=+2468.892395552" watchObservedRunningTime="2026-10-30 22:27:28.594301487 +0000 UTC m=+2468.893711367" Oct 30 22:27:28.794595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:28.794561 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:29.801122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:29.801040 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:30.493921 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.493881 4991 generic.go:356] "Generic (PLEG): container finished" podID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerID="1896233dfe3fbd615365702b549826b6f91a9d2b726259f779166851dd657784" exitCode=0 Oct 30 22:27:30.494570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.494130 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" event={"ID":"0c0e72d9-7a32-4f56-b382-c998e1bf046f","Type":"ContainerDied","Data":"1896233dfe3fbd615365702b549826b6f91a9d2b726259f779166851dd657784"} Oct 30 22:27:30.494570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.494233 4991 scope.go:122] "RemoveContainer" containerID="426319d52a16627dfa1ba4fa21727cb49d8f386e3dac7537b6c50bd756761994" Oct 30 22:27:30.494570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.494347 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:30.497089 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.495666 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:27:30.588565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.588543 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:27:30.651358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.650588 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-credential-socket\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.651358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.650843 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-credential-socket" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:30.651358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.650950 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istiod-ca-cert\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.651358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.651299 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istiod-ca-cert" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:30.651669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.651393 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-kube-api-access-67rdt\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-kube-api-access-67rdt\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.651754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.651734 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-data\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.652073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.652038 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-data" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:30.652179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.652093 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-socket\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.652395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.652373 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-socket" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:30.652498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.652434 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-token\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-token\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.652850 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.652829 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-podinfo\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.652948 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.652883 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-certs\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.652948 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.652914 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-envoy\") pod \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\" (UID: \"0c0e72d9-7a32-4f56-b382-c998e1bf046f\") " Oct 30 22:27:30.653400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.653369 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.653400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.653394 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.653557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.653410 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.653557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.653424 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.654057 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.654027 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-certs" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:30.654148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.654079 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-kube-api-access-67rdt" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "kube-api-access-67rdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:30.654871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.654846 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-token" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:30.656983 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.656960 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-envoy" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:30.657214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.657193 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-podinfo" pod "0c0e72d9-7a32-4f56-b382-c998e1bf046f" (UID: "0c0e72d9-7a32-4f56-b382-c998e1bf046f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:30.763348 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.757906 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rdt\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-kube-api-access-67rdt\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.763642 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.763617 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.763787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.763772 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.763879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.763868 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.763977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:30.763965 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0e72d9-7a32-4f56-b382-c998e1bf046f-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:31.501510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:31.501482 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" Oct 30 22:27:31.502983 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:31.502954 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq" event={"ID":"0c0e72d9-7a32-4f56-b382-c998e1bf046f","Type":"ContainerDied","Data":"7ea11a93e94566a8a1c572397558456515fbfe38f8909f108bbcd2646279e0f3"} Oct 30 22:27:31.503140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:31.503016 4991 scope.go:122] "RemoveContainer" containerID="1896233dfe3fbd615365702b549826b6f91a9d2b726259f779166851dd657784" Oct 30 22:27:31.713904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:31.713864 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq"] Oct 30 22:27:31.753392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:31.752677 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a3357-openshift-default-664b5474c9-2mmgq"] Oct 30 22:27:31.875700 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:31.875668 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" path="/var/lib/kubelet/pods/0c0e72d9-7a32-4f56-b382-c998e1bf046f/volumes" Oct 30 22:27:36.585388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:36.585351 4991 generic.go:356] "Generic (PLEG): container finished" podID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerID="0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b" exitCode=0 Oct 30 22:27:36.585960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:36.585421 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" event={"ID":"0b13dea8-9817-452c-a658-ce355e4b0de6","Type":"ContainerDied","Data":"0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b"} Oct 30 22:27:36.585960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:36.585844 4991 scope.go:122] "RemoveContainer" containerID="0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b" Oct 30 22:27:36.587472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:36.587431 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:36.587600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:36.587519 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:36.587600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:36.587565 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:37.593098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:37.593061 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" event={"ID":"0b13dea8-9817-452c-a658-ce355e4b0de6","Type":"ContainerStarted","Data":"1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751"} Oct 30 22:27:37.761303 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:37.761263 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:27:37.762408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:37.762384 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:27:37.762511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:37.762430 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:27:38.690881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:38.690852 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:27:38.761617 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:38.761585 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:27:38.761757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:38.761649 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:27:39.761979 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:39.761940 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:27:39.762426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:39.762001 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:27:40.761964 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:40.761921 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:27:40.762155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:40.761975 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:27:41.761798 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:41.761761 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:27:41.762010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:41.761818 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 22:27:42.872154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:42.872117 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:27:42.874394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:42.874366 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:27:46.657206 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.657160 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8"] Oct 30 22:27:46.657721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.657704 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" Oct 30 22:27:46.657768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.657722 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" Oct 30 22:27:46.657841 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.657831 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" Oct 30 22:27:46.657876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.657844 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" Oct 30 22:27:46.661196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.661176 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.710751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.710724 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e524ce-openshift-default-dockercfg-zs996\"" Oct 30 22:27:46.761356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.761308 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8"] Oct 30 22:27:46.761878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.761842 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767c2\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-kube-api-access-767c2\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.761995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.761894 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.761995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.761947 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.762121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.762084 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.762185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.762159 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.762370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.762315 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.762477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.762416 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.762559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.762485 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c585d34-7460-440f-bbb4-c00a776e52ba-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.762559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.762527 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.863682 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863648 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.863846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863692 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.863846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863740 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c585d34-7460-440f-bbb4-c00a776e52ba-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.863846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863776 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863868 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767c2\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-kube-api-access-767c2\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863899 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863929 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863959 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.863996 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.864157 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.864372 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.864409 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.864450 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.864647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.864621 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c585d34-7460-440f-bbb4-c00a776e52ba-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.865994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.865972 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.866529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.866511 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.921235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.921179 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.921584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.921568 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767c2\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-kube-api-access-767c2\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:46.975025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:46.974998 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:47.416134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.416098 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8"] Oct 30 22:27:47.423065 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:27:47.423029 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c585d34_7460_440f_bbb4_c00a776e52ba.slice/crio-9ac55a05a2cafc501543623fc3dd252444862b7542e5a8cb5e3e3c8e6d08cdcb WatchSource:0}: Error finding container 9ac55a05a2cafc501543623fc3dd252444862b7542e5a8cb5e3e3c8e6d08cdcb: Status 404 returned error can't find the container with id 9ac55a05a2cafc501543623fc3dd252444862b7542e5a8cb5e3e3c8e6d08cdcb Oct 30 22:27:47.425518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.425488 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:47.425615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.425551 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:47.425615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.425580 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:27:47.683721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.683636 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" event={"ID":"0c585d34-7460-440f-bbb4-c00a776e52ba","Type":"ContainerStarted","Data":"172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46"} Oct 30 22:27:47.683721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.683699 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" event={"ID":"0c585d34-7460-440f-bbb4-c00a776e52ba","Type":"ContainerStarted","Data":"9ac55a05a2cafc501543623fc3dd252444862b7542e5a8cb5e3e3c8e6d08cdcb"} Oct 30 22:27:47.801028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.800966 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podStartSLOduration=1.800949051 podStartE2EDuration="1.800949051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:47.798468742 +0000 UTC m=+2488.097878641" watchObservedRunningTime="2026-10-30 22:27:47.800949051 +0000 UTC m=+2488.100358932" Oct 30 22:27:47.917801 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:27:47.917761 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:27:47.975817 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:47.975789 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:48.977689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:48.977651 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:48.978080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:48.977719 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:49.976207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:49.976169 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:49.976438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:49.976226 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:50.975749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:50.975710 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:50.976158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:50.975764 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:51.976381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:51.976349 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:51.976846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:51.976819 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:52.976291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:52.976250 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:52.976625 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:52.976308 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:53.975465 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:53.975431 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:53.975648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:53.975494 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:54.980977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:54.980947 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:54.981448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:54.981423 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:27:54.982061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:27:54.982033 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:28:17.913050 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:28:17.913009 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:28:18.666116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:18.666082 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz"] Oct 30 22:28:18.666395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:18.666348 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" containerID="cri-o://9c392279c9584f67d0c600ed2a35ff4b313d0f2ae53a06d1d0dee7d92522c348" gracePeriod=30 Oct 30 22:28:23.963281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:23.963190 4991 generic.go:356] "Generic (PLEG): container finished" podID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerID="9c392279c9584f67d0c600ed2a35ff4b313d0f2ae53a06d1d0dee7d92522c348" exitCode=0 Oct 30 22:28:23.963821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:23.963476 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" event={"ID":"ef9c61f6-a801-4f19-8d90-44cf2d436c60","Type":"ContainerDied","Data":"9c392279c9584f67d0c600ed2a35ff4b313d0f2ae53a06d1d0dee7d92522c348"} Oct 30 22:28:23.989105 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:23.989083 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151116 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-kube-api-access-25wng\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-kube-api-access-25wng\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151221 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-data\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151278 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-podinfo\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151308 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-envoy\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151426 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-socket\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151485 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-certs\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151526 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-token\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151599 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-credential-socket\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.152344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.151652 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istiod-ca-cert\") pod \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\" (UID: \"ef9c61f6-a801-4f19-8d90-44cf2d436c60\") " Oct 30 22:28:24.153717 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.153687 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-socket" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.154533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.153992 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-certs" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.154533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.154501 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-credential-socket" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.155333 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.155284 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-data" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.155902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.155867 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istiod-ca-cert" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:24.159503 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.159460 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-podinfo" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:24.166591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.166557 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-envoy" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.167104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.167068 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-token" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:24.168568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.168543 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-kube-api-access-25wng" pod "ef9c61f6-a801-4f19-8d90-44cf2d436c60" (UID: "ef9c61f6-a801-4f19-8d90-44cf2d436c60"). InnerVolumeSpecName "kube-api-access-25wng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253429 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253468 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253485 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253499 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253514 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wng\" (UniqueName: \"kubernetes.io/projected/ef9c61f6-a801-4f19-8d90-44cf2d436c60-kube-api-access-25wng\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253529 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253543 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253556 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.253612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.253569 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c61f6-a801-4f19-8d90-44cf2d436c60-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.973657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.973625 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" Oct 30 22:28:24.973657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.973646 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz" event={"ID":"ef9c61f6-a801-4f19-8d90-44cf2d436c60","Type":"ContainerDied","Data":"146d9eed289204d3fc0aa3c14aa310b25e984d6cad1c5642dbd446e37d5552e0"} Oct 30 22:28:24.974103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:24.973723 4991 scope.go:122] "RemoveContainer" containerID="9c392279c9584f67d0c600ed2a35ff4b313d0f2ae53a06d1d0dee7d92522c348" Oct 30 22:28:25.178074 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:25.178031 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz"] Oct 30 22:28:25.250391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:25.249515 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-cn2dz"] Oct 30 22:28:25.874852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:25.874819 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" path="/var/lib/kubelet/pods/ef9c61f6-a801-4f19-8d90-44cf2d436c60/volumes" Oct 30 22:28:28.058595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.058550 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh"] Oct 30 22:28:28.059387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.059357 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" Oct 30 22:28:28.059387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.059384 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" Oct 30 22:28:28.059584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.059403 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" Oct 30 22:28:28.059584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.059411 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0e72d9-7a32-4f56-b382-c998e1bf046f" containerName="istio-proxy" Oct 30 22:28:28.059662 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.059594 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9c61f6-a801-4f19-8d90-44cf2d436c60" containerName="istio-proxy" Oct 30 22:28:28.063473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.063450 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.114893 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.114850 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b47e8d-openshift-default-dockercfg-bxx6t\"" Oct 30 22:28:28.165943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.165911 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh"] Oct 30 22:28:28.203112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203070 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203167 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203220 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203255 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203516 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203460 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09394e12-bb87-4a21-891a-5fdae7e11eb1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203558 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84jv4\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-kube-api-access-84jv4\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203606 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203632 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.203689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.203667 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.304922 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.304883 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.304922 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.304929 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.304964 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305028 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09394e12-bb87-4a21-891a-5fdae7e11eb1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305103 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84jv4\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-kube-api-access-84jv4\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305143 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305166 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305195 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305255 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305457 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305576 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.305786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.305607 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.306078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.306036 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09394e12-bb87-4a21-891a-5fdae7e11eb1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.306280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.306256 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.307570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.307547 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.307837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.307813 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.363578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.363499 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.364049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.364025 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84jv4\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-kube-api-access-84jv4\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.378471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.378435 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:28.828227 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.828182 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh"] Oct 30 22:28:28.833830 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:28:28.833801 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09394e12_bb87_4a21_891a_5fdae7e11eb1.slice/crio-65b96fad1b46bd28e705f1dc57239c94bfd206797445f524d7e48ba2e5b95fa4 WatchSource:0}: Error finding container 65b96fad1b46bd28e705f1dc57239c94bfd206797445f524d7e48ba2e5b95fa4: Status 404 returned error can't find the container with id 65b96fad1b46bd28e705f1dc57239c94bfd206797445f524d7e48ba2e5b95fa4 Oct 30 22:28:28.836314 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.836282 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:28.836455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.836374 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:28.836455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:28.836410 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:29.006487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:29.006444 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" event={"ID":"09394e12-bb87-4a21-891a-5fdae7e11eb1","Type":"ContainerStarted","Data":"d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26"} Oct 30 22:28:29.006669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:29.006507 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" event={"ID":"09394e12-bb87-4a21-891a-5fdae7e11eb1","Type":"ContainerStarted","Data":"65b96fad1b46bd28e705f1dc57239c94bfd206797445f524d7e48ba2e5b95fa4"} Oct 30 22:28:29.121418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:29.121230 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podStartSLOduration=1.121212472 podStartE2EDuration="1.121212472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:29.121177216 +0000 UTC m=+2529.420587124" watchObservedRunningTime="2026-10-30 22:28:29.121212472 +0000 UTC m=+2529.420622359" Oct 30 22:28:29.379509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:29.379429 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:29.380705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:29.380672 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:28:29.380836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:29.380728 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:28:30.379859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:30.379431 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:28:30.379859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:30.379506 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:28:31.378990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:31.378904 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:28:31.378990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:31.378972 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:28:32.051873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.051841 4991 generic.go:356] "Generic (PLEG): container finished" podID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerID="a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927" exitCode=0 Oct 30 22:28:32.052280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.051913 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" event={"ID":"d199c57d-49ec-4b6b-8cee-6511796ee6de","Type":"ContainerDied","Data":"a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927"} Oct 30 22:28:32.052362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.052348 4991 scope.go:122] "RemoveContainer" containerID="a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927" Oct 30 22:28:32.053775 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.053749 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:32.053852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.053815 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:32.053897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.053853 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:32.379134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.379056 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:28:32.379134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:32.379112 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:28:33.099622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:33.098785 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" event={"ID":"d199c57d-49ec-4b6b-8cee-6511796ee6de","Type":"ContainerStarted","Data":"be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088"} Oct 30 22:28:33.397411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:33.396566 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:28:33.397411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:33.396733 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:28:33.828353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:33.823634 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:28:33.828353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:33.823746 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:28:33.828353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:33.823787 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 22:28:34.380159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:34.379796 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:28:34.380159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:34.379855 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:28:34.831344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:34.823676 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:28:34.831344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:34.823739 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 22:28:35.379354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:35.379293 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:28:35.379546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:35.379386 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:28:35.824345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:35.823711 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:28:35.824345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:35.823776 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 22:28:36.691979 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:36.691936 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:36.692464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:36.692430 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:36.693255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:36.693234 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:28:36.832345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:36.825647 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:28:36.832345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:36.825703 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 22:28:37.835847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:37.835810 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:28:37.836552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:37.836520 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 22:28:38.827692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:38.827661 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:28:38.827959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:38.827937 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:28:38.828579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:38.828559 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:28:47.158923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.158884 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb"] Oct 30 22:28:47.165247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.165225 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.202372 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.202340 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb"] Oct 30 22:28:47.268991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.268948 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.269256 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.269232 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.269433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.269411 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.269566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.269550 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718a5205-8ad4-44a5-9529-cc6b5fee8737-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.269715 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.269699 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.282504 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.282476 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.290043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.290013 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.290368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.290343 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxjn\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-kube-api-access-4rxjn\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.290600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.290579 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393685 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393650 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxjn\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-kube-api-access-4rxjn\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393697 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393725 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393754 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393777 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393797 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718a5205-8ad4-44a5-9529-cc6b5fee8737-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393832 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.393888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393882 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.394335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.393932 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.394985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.394948 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.395900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.395729 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.396033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.395786 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718a5205-8ad4-44a5-9529-cc6b5fee8737-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.396144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.396118 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.396348 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.396124 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.398010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.397987 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.398268 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.398249 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.469643 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.467666 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.478364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.478306 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxjn\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-kube-api-access-4rxjn\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.481339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.481081 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:47.912511 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:28:47.912473 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:28:47.997637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:47.997591 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb"] Oct 30 22:28:48.005984 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:28:48.005958 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718a5205_8ad4_44a5_9529_cc6b5fee8737.slice/crio-d3b97c6a120ea7e012249ecf2cd4c39d38eed94218552ad34385debfdf9587ef WatchSource:0}: Error finding container d3b97c6a120ea7e012249ecf2cd4c39d38eed94218552ad34385debfdf9587ef: Status 404 returned error can't find the container with id d3b97c6a120ea7e012249ecf2cd4c39d38eed94218552ad34385debfdf9587ef Oct 30 22:28:48.009513 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:48.009466 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:48.009592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:48.009545 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:48.009592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:48.009585 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:48.248339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:48.245821 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" event={"ID":"718a5205-8ad4-44a5-9529-cc6b5fee8737","Type":"ContainerStarted","Data":"54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec"} Oct 30 22:28:48.248339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:48.245892 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" event={"ID":"718a5205-8ad4-44a5-9529-cc6b5fee8737","Type":"ContainerStarted","Data":"d3b97c6a120ea7e012249ecf2cd4c39d38eed94218552ad34385debfdf9587ef"} Oct 30 22:28:48.386346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:48.385355 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podStartSLOduration=1.385335652 podStartE2EDuration="1.385335652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:48.3828269 +0000 UTC m=+2548.682236783" watchObservedRunningTime="2026-10-30 22:28:48.385335652 +0000 UTC m=+2548.684745535" Oct 30 22:28:48.482426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:48.482389 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:49.482628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:49.482592 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:28:49.483037 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:49.482642 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:28:50.482343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:50.481879 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:28:50.482343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:50.481960 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:28:51.483385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:51.482188 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:28:51.483385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:51.482263 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:28:52.482343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:52.481820 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:28:52.482343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:52.481885 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:28:53.482518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:53.482479 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:28:53.482923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:53.482541 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:28:54.482384 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:54.482339 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:28:54.482612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:54.482426 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:28:55.864816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:55.864779 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:55.885568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:55.885289 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:55.885568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:55.885396 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:28:57.341355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:57.341307 4991 generic.go:356] "Generic (PLEG): container finished" podID="9e698f07-79f2-451e-8777-e73f923096c1" containerID="15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422" exitCode=0 Oct 30 22:28:57.341355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:57.341349 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" event={"ID":"9e698f07-79f2-451e-8777-e73f923096c1","Type":"ContainerDied","Data":"15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422"} Oct 30 22:28:57.341819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:57.341720 4991 scope.go:122] "RemoveContainer" containerID="15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422" Oct 30 22:28:57.343144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:57.343118 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:57.343232 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:57.343190 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:57.343232 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:57.343225 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:28:58.354605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.354572 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" event={"ID":"9e698f07-79f2-451e-8777-e73f923096c1","Type":"ContainerStarted","Data":"88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2"} Oct 30 22:28:58.367157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.367126 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c"] Oct 30 22:28:58.367337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.367304 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" podUID="008e673b-52fc-4700-a2ec-769a25faa5f6" containerName="discovery" containerID="cri-o://2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e" gracePeriod=30 Oct 30 22:28:58.674781 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.674754 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:28:58.795382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.795349 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:28:58.796220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.796193 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:28:58.796379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.796256 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:28:58.818572 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.818547 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-dns-cert\") pod \"008e673b-52fc-4700-a2ec-769a25faa5f6\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " Oct 30 22:28:58.818670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.818611 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-token\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-token\") pod \"008e673b-52fc-4700-a2ec-769a25faa5f6\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " Oct 30 22:28:58.818726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.818687 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-ca-configmap\") pod \"008e673b-52fc-4700-a2ec-769a25faa5f6\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " Oct 30 22:28:58.818763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.818735 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008e673b-52fc-4700-a2ec-769a25faa5f6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/008e673b-52fc-4700-a2ec-769a25faa5f6-local-certs\") pod \"008e673b-52fc-4700-a2ec-769a25faa5f6\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " Oct 30 22:28:58.818797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.818783 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-kube-api-access-hxmbm\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-kube-api-access-hxmbm\") pod \"008e673b-52fc-4700-a2ec-769a25faa5f6\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " Oct 30 22:28:58.818838 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.818814 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-cacerts\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-cacerts\") pod \"008e673b-52fc-4700-a2ec-769a25faa5f6\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " Oct 30 22:28:58.818932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.818915 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-kubeconfig\") pod \"008e673b-52fc-4700-a2ec-769a25faa5f6\" (UID: \"008e673b-52fc-4700-a2ec-769a25faa5f6\") " Oct 30 22:28:58.819189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.819082 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-ca-configmap" pod "008e673b-52fc-4700-a2ec-769a25faa5f6" (UID: "008e673b-52fc-4700-a2ec-769a25faa5f6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:58.846401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.819406 4991 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-ca-configmap\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:58.855594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.849269 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008e673b-52fc-4700-a2ec-769a25faa5f6-local-certs" pod "008e673b-52fc-4700-a2ec-769a25faa5f6" (UID: "008e673b-52fc-4700-a2ec-769a25faa5f6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:58.855594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.854560 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-dns-cert" pod "008e673b-52fc-4700-a2ec-769a25faa5f6" (UID: "008e673b-52fc-4700-a2ec-769a25faa5f6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:58.857470 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.857438 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-token" pod "008e673b-52fc-4700-a2ec-769a25faa5f6" (UID: "008e673b-52fc-4700-a2ec-769a25faa5f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:58.857815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.857758 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-cacerts" pod "008e673b-52fc-4700-a2ec-769a25faa5f6" (UID: "008e673b-52fc-4700-a2ec-769a25faa5f6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:58.857815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.857776 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-kubeconfig" pod "008e673b-52fc-4700-a2ec-769a25faa5f6" (UID: "008e673b-52fc-4700-a2ec-769a25faa5f6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:58.857919 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.857845 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-kube-api-access-hxmbm" pod "008e673b-52fc-4700-a2ec-769a25faa5f6" (UID: "008e673b-52fc-4700-a2ec-769a25faa5f6"). InnerVolumeSpecName "kube-api-access-hxmbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:58.945616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.943171 4991 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/008e673b-52fc-4700-a2ec-769a25faa5f6-local-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:58.945616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.943214 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxmbm\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-kube-api-access-hxmbm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:58.945616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.943230 4991 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-cacerts\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:58.945616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.943243 4991 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-kubeconfig\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:58.945616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.943260 4991 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-csr-dns-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:58.945616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:58.943274 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008e673b-52fc-4700-a2ec-769a25faa5f6-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.363410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.363373 4991 generic.go:356] "Generic (PLEG): container finished" podID="008e673b-52fc-4700-a2ec-769a25faa5f6" containerID="2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e" exitCode=0 Oct 30 22:28:59.363882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.363422 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" Oct 30 22:28:59.363882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.363446 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" event={"ID":"008e673b-52fc-4700-a2ec-769a25faa5f6","Type":"ContainerDied","Data":"2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e"} Oct 30 22:28:59.363882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.363497 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c" event={"ID":"008e673b-52fc-4700-a2ec-769a25faa5f6","Type":"ContainerDied","Data":"cf2d29847a4adc475511100022ce54ff5668a5ea5e4a1a5f947d047985101627"} Oct 30 22:28:59.363882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.363518 4991 scope.go:122] "RemoveContainer" containerID="2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e" Oct 30 22:28:59.374553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.374311 4991 scope.go:122] "RemoveContainer" containerID="2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e" Oct 30 22:28:59.374611 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:28:59.374552 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e\": container with ID starting with 2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e not found: ID does not exist" containerID="2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e" Oct 30 22:28:59.374611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.374577 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e"} err="failed to get container status \"2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e\": rpc error: code = NotFound desc = could not find container \"2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e\": container with ID starting with 2b81d122bf829e8e46393545d45766b31c83124d4d3b34f9f3ae9fbbdaab6c4e not found: ID does not exist" Oct 30 22:28:59.533344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.529510 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c"] Oct 30 22:28:59.567075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.567041 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qq6c"] Oct 30 22:28:59.795560 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.795487 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:28:59.795560 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.795536 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:28:59.875727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:28:59.875695 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008e673b-52fc-4700-a2ec-769a25faa5f6" path="/var/lib/kubelet/pods/008e673b-52fc-4700-a2ec-769a25faa5f6/volumes" Oct 30 22:29:00.495342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:00.495276 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:29:00.795121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:00.795035 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:29:00.795121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:00.795104 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:29:01.795297 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:01.795255 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:29:01.795811 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:01.795342 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:29:02.794900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:02.794863 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:29:02.795116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:02.795094 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:29:03.798976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:03.798945 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:29:03.800099 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:03.800078 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:29:04.057910 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.057831 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx"] Oct 30 22:29:04.058089 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.058065 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" podUID="e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" containerName="maas-api" containerID="cri-o://dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab" gracePeriod=30 Oct 30 22:29:04.336641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.336567 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:29:04.398495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.398454 4991 generic.go:356] "Generic (PLEG): container finished" podID="e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" containerID="dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab" exitCode=0 Oct 30 22:29:04.398667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.398499 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" Oct 30 22:29:04.398667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.398515 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" event={"ID":"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63","Type":"ContainerDied","Data":"dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab"} Oct 30 22:29:04.398667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.398577 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx" event={"ID":"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63","Type":"ContainerDied","Data":"8f22ea1e83c48269dec3567de5308b5d4385ef0ad17ffcab5400839ed8230e66"} Oct 30 22:29:04.398667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.398610 4991 scope.go:122] "RemoveContainer" containerID="dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab" Oct 30 22:29:04.407953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.407934 4991 scope.go:122] "RemoveContainer" containerID="dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab" Oct 30 22:29:04.408243 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:04.408222 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab\": container with ID starting with dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab not found: ID does not exist" containerID="dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab" Oct 30 22:29:04.408305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.408254 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab"} err="failed to get container status \"dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab\": rpc error: code = NotFound desc = could not find container \"dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab\": container with ID starting with dc226c01424450c0ca5bdb0581bfc2753da613ea936f94c79cb84462d3670aab not found: ID does not exist" Oct 30 22:29:04.409509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.409488 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls\") pod \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " Oct 30 22:29:04.409596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.409561 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-openshift-service-ca\") pod \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " Oct 30 22:29:04.409738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.409725 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-metrics-certs\") pod \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " Oct 30 22:29:04.409799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.409772 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-kube-api-access-p4nnk\" (UniqueName: \"kubernetes.io/projected/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-kube-api-access-p4nnk\") pod \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\" (UID: \"e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63\") " Oct 30 22:29:04.409920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.409897 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-openshift-service-ca" pod "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" (UID: "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:04.410374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.410350 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:04.411880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.411857 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-kube-api-access-p4nnk" pod "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" (UID: "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63"). InnerVolumeSpecName "kube-api-access-p4nnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:04.411980 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.411933 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls" pod "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" (UID: "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:04.412048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.412024 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-metrics-certs" pod "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" (UID: "e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:04.511890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.511861 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:04.511890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.511888 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4nnk\" (UniqueName: \"kubernetes.io/projected/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-kube-api-access-p4nnk\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:04.511890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.511898 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:04.815676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.815637 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx"] Oct 30 22:29:04.822153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:04.822123 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf5019-6db4fdfb9c-rsngx"] Oct 30 22:29:05.045208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.045179 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv"] Oct 30 22:29:05.045736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.045720 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" containerName="maas-api" Oct 30 22:29:05.045778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.045738 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" containerName="maas-api" Oct 30 22:29:05.045778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.045754 4991 cpu_manager.go:414] "containerMap: removing container" podUID="008e673b-52fc-4700-a2ec-769a25faa5f6" containerName="discovery" Oct 30 22:29:05.045778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.045759 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e673b-52fc-4700-a2ec-769a25faa5f6" containerName="discovery" Oct 30 22:29:05.045879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.045852 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" containerName="maas-api" Oct 30 22:29:05.045879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.045864 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="008e673b-52fc-4700-a2ec-769a25faa5f6" containerName="discovery" Oct 30 22:29:05.048818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.048801 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.112382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.110501 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-28f124-openshift-default-dockercfg-q9xb5\"" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.116836 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv"] Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.122970 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-token\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123019 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-data\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123065 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-certs\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123123 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-envoy\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123167 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgcx\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-kube-api-access-9sgcx\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123195 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-podinfo\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123223 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-credential-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123271 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istiod-ca-cert\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.124340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.123420 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.224893 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.224851 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-token\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.224903 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-data\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.224942 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-certs\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225093 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-envoy\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225158 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgcx\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-kube-api-access-9sgcx\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225188 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-podinfo\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225218 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-credential-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225260 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istiod-ca-cert\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225465 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225398 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225530 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-data\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225840 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-certs\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.225859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.225847 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.226437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.226412 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-credential-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.227227 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.227196 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istiod-ca-cert\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.228707 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.228572 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-envoy\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.230065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.230044 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-podinfo\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.284394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.284365 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgcx\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-kube-api-access-9sgcx\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.285708 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.285685 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-token\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.361476 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.361392 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:05.793204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.793172 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv"] Oct 30 22:29:05.800188 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:05.800160 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0ebfa3_73bf_4704_9c0d_65b0d9e0267c.slice/crio-efafc3d564447c39f3f24f8b0660bfb3c5b108fcab3278f23279abb3306d4606 WatchSource:0}: Error finding container efafc3d564447c39f3f24f8b0660bfb3c5b108fcab3278f23279abb3306d4606: Status 404 returned error can't find the container with id efafc3d564447c39f3f24f8b0660bfb3c5b108fcab3278f23279abb3306d4606 Oct 30 22:29:05.802544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.802502 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:05.802648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.802584 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:05.802648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.802630 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:05.876671 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:05.876640 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63" path="/var/lib/kubelet/pods/e38c7c9a-d6bf-4ccc-90e6-3a5cff33ef63/volumes" Oct 30 22:29:06.415000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:06.414961 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" event={"ID":"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c","Type":"ContainerStarted","Data":"7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27"} Oct 30 22:29:06.415240 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:06.415029 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" event={"ID":"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c","Type":"ContainerStarted","Data":"efafc3d564447c39f3f24f8b0660bfb3c5b108fcab3278f23279abb3306d4606"} Oct 30 22:29:06.539447 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:06.539384 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podStartSLOduration=1.539366946 podStartE2EDuration="1.539366946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:06.537004671 +0000 UTC m=+2566.836414605" watchObservedRunningTime="2026-10-30 22:29:06.539366946 +0000 UTC m=+2566.838776825" Oct 30 22:29:07.362035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:07.362001 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:07.367410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:07.367382 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:07.421413 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:07.421388 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:07.422204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:07.422187 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:29:13.063013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.062969 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr"] Oct 30 22:29:13.066728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.066704 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.117279 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.117246 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72224bc6-33bb-43a2-bd01-df0109a97984-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.117452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.117421 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdx6t\" (UniqueName: \"kubernetes.io/projected/72224bc6-33bb-43a2-bd01-df0109a97984-kube-api-access-zdx6t\") pod \"payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.123033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.123003 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr"] Oct 30 22:29:13.199340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.166810 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-28f124-dockercfg-7vk5z\"" Oct 30 22:29:13.205585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.200412 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-28f124\"" Oct 30 22:29:13.218488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.217896 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9"] Oct 30 22:29:13.218926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.218901 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72224bc6-33bb-43a2-bd01-df0109a97984-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.219048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.219023 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdx6t\" (UniqueName: \"kubernetes.io/projected/72224bc6-33bb-43a2-bd01-df0109a97984-kube-api-access-zdx6t\") pod \"payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.220196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.220172 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72224bc6-33bb-43a2-bd01-df0109a97984-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.223676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.223656 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.265174 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.265147 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9"] Oct 30 22:29:13.277190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.277164 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdx6t\" (UniqueName: \"kubernetes.io/projected/72224bc6-33bb-43a2-bd01-df0109a97984-kube-api-access-zdx6t\") pod \"payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.320823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.320756 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94973040-d19d-4a9a-8b45-b8cdd3bfd176-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.320823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.320791 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzrs\" (UniqueName: \"kubernetes.io/projected/94973040-d19d-4a9a-8b45-b8cdd3bfd176-kube-api-access-xdzrs\") pod \"payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.382235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.382204 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:13.422510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.422476 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94973040-d19d-4a9a-8b45-b8cdd3bfd176-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.422658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.422522 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzrs\" (UniqueName: \"kubernetes.io/projected/94973040-d19d-4a9a-8b45-b8cdd3bfd176-kube-api-access-xdzrs\") pod \"payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.423229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.423208 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94973040-d19d-4a9a-8b45-b8cdd3bfd176-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.446375 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.445444 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz"] Oct 30 22:29:13.451439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.450683 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.496146 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.496110 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzrs\" (UniqueName: \"kubernetes.io/projected/94973040-d19d-4a9a-8b45-b8cdd3bfd176-kube-api-access-xdzrs\") pod \"payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.511402 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.511371 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz"] Oct 30 22:29:13.524100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524072 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524101 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea52d3c2-640b-420d-a955-12544c223fbc-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524128 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea52d3c2-640b-420d-a955-12544c223fbc-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524233 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524284 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgn45\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-kube-api-access-rgn45\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524351 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524407 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524487 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.524658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.524563 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.537897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.537863 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:13.626353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626307 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea52d3c2-640b-420d-a955-12544c223fbc-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626378 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626424 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn45\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-kube-api-access-rgn45\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626493 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626518 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626579 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626807 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626694 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626807 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626741 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626807 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626782 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea52d3c2-640b-420d-a955-12544c223fbc-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.626955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.626919 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.627169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.627145 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.627395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.627371 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.627537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.627490 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea52d3c2-640b-420d-a955-12544c223fbc-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.627738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.627709 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.630564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.630536 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea52d3c2-640b-420d-a955-12544c223fbc-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.630711 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.630691 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.684130 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.684104 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn45\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-kube-api-access-rgn45\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.684357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.684312 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.773712 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.773683 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr"] Oct 30 22:29:13.775004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.774987 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:13.780561 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:13.780529 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72224bc6_33bb_43a2_bd01_df0109a97984.slice/crio-5391eb784895399f9a4427f30479d0e328c10965d4b368abe7ad2ebf97e6c208 WatchSource:0}: Error finding container 5391eb784895399f9a4427f30479d0e328c10965d4b368abe7ad2ebf97e6c208: Status 404 returned error can't find the container with id 5391eb784895399f9a4427f30479d0e328c10965d4b368abe7ad2ebf97e6c208 Oct 30 22:29:13.926903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:13.926725 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9"] Oct 30 22:29:13.933866 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:13.933833 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94973040_d19d_4a9a_8b45_b8cdd3bfd176.slice/crio-da00492e0f49686b258bd94e91b9afc9a58c6701efedd86bc01dab1a9b4bcf83 WatchSource:0}: Error finding container da00492e0f49686b258bd94e91b9afc9a58c6701efedd86bc01dab1a9b4bcf83: Status 404 returned error can't find the container with id da00492e0f49686b258bd94e91b9afc9a58c6701efedd86bc01dab1a9b4bcf83 Oct 30 22:29:14.047179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.047147 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr"] Oct 30 22:29:14.047408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.047362 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" containerID="cri-o://1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751" gracePeriod=30 Oct 30 22:29:14.188432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.178385 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz"] Oct 30 22:29:14.189712 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:14.189679 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea52d3c2_640b_420d_a955_12544c223fbc.slice/crio-6da43fc37d296e0369ca6cd3a868d84c2222a873cbaabe9bb33249b166633d3f WatchSource:0}: Error finding container 6da43fc37d296e0369ca6cd3a868d84c2222a873cbaabe9bb33249b166633d3f: Status 404 returned error can't find the container with id 6da43fc37d296e0369ca6cd3a868d84c2222a873cbaabe9bb33249b166633d3f Oct 30 22:29:14.193195 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.193154 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:14.193288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.193251 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:14.193378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.193297 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:14.494344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.485823 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" event={"ID":"ea52d3c2-640b-420d-a955-12544c223fbc","Type":"ContainerStarted","Data":"55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94"} Oct 30 22:29:14.494344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.485890 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" event={"ID":"ea52d3c2-640b-420d-a955-12544c223fbc","Type":"ContainerStarted","Data":"6da43fc37d296e0369ca6cd3a868d84c2222a873cbaabe9bb33249b166633d3f"} Oct 30 22:29:14.494344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.488635 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" event={"ID":"94973040-d19d-4a9a-8b45-b8cdd3bfd176","Type":"ContainerStarted","Data":"da00492e0f49686b258bd94e91b9afc9a58c6701efedd86bc01dab1a9b4bcf83"} Oct 30 22:29:14.494344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.490860 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" event={"ID":"72224bc6-33bb-43a2-bd01-df0109a97984","Type":"ContainerStarted","Data":"5391eb784895399f9a4427f30479d0e328c10965d4b368abe7ad2ebf97e6c208"} Oct 30 22:29:14.606259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.606181 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podStartSLOduration=1.606161691 podStartE2EDuration="1.606161691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:14.602117184 +0000 UTC m=+2574.901527079" watchObservedRunningTime="2026-10-30 22:29:14.606161691 +0000 UTC m=+2574.905571570" Oct 30 22:29:14.775347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:14.775239 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:15.817751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:15.815312 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:29:15.817751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:15.815414 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:29:16.775684 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:16.775644 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:29:16.775856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:16.775703 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:29:17.776603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:17.776350 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:29:17.776603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:17.776407 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:29:17.916739 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:17.916699 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:29:18.776706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:18.776356 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:29:18.776706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:18.776403 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:29:19.469021 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.468677 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:29:19.493781 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.493754 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-credential-socket\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.493938 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.493793 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-envoy\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.493938 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.493818 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-token\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-token\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.493938 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.493874 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-podinfo\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.493938 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.493935 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-data\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.494353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.493971 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-socket\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.494353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.493999 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b13dea8-9817-452c-a658-ce355e4b0de6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13dea8-9817-452c-a658-ce355e4b0de6-istiod-ca-cert\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.494353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.494026 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-certs\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.494353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.494059 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-kube-api-access-vs4lj\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-kube-api-access-vs4lj\") pod \"0b13dea8-9817-452c-a658-ce355e4b0de6\" (UID: \"0b13dea8-9817-452c-a658-ce355e4b0de6\") " Oct 30 22:29:19.495869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.495518 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b13dea8-9817-452c-a658-ce355e4b0de6-istiod-ca-cert" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:19.495869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.495539 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-certs" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:19.495869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.495632 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-socket" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:19.495869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.495773 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-credential-socket" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:19.496196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.495959 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-data" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:19.496757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.496735 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-podinfo" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:19.497237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.497220 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-token" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:19.497453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.497430 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-kube-api-access-vs4lj" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "kube-api-access-vs4lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:19.497982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.497962 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-envoy" pod "0b13dea8-9817-452c-a658-ce355e4b0de6" (UID: "0b13dea8-9817-452c-a658-ce355e4b0de6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:19.586413 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.586374 4991 generic.go:356] "Generic (PLEG): container finished" podID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerID="1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751" exitCode=0 Oct 30 22:29:19.586556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.586432 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" event={"ID":"0b13dea8-9817-452c-a658-ce355e4b0de6","Type":"ContainerDied","Data":"1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751"} Oct 30 22:29:19.586556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.586483 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" event={"ID":"0b13dea8-9817-452c-a658-ce355e4b0de6","Type":"ContainerDied","Data":"fb689fe2d7a56dbca6f7d14b6e2a52411f5df60ff196860aecb23fc411e570dc"} Oct 30 22:29:19.586556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.586505 4991 scope.go:122] "RemoveContainer" containerID="1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751" Oct 30 22:29:19.586670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.586636 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594899 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594922 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594933 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594942 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594951 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594960 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13dea8-9817-452c-a658-ce355e4b0de6-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594969 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594977 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4lj\" (UniqueName: \"kubernetes.io/projected/0b13dea8-9817-452c-a658-ce355e4b0de6-kube-api-access-vs4lj\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.595002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.594986 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13dea8-9817-452c-a658-ce355e4b0de6-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:19.617630 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.617611 4991 scope.go:122] "RemoveContainer" containerID="0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b" Oct 30 22:29:19.678531 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.678503 4991 scope.go:122] "RemoveContainer" containerID="1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751" Oct 30 22:29:19.678817 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:19.678795 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751\": container with ID starting with 1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751 not found: ID does not exist" containerID="1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751" Oct 30 22:29:19.678895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.678826 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751"} err="failed to get container status \"1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751\": rpc error: code = NotFound desc = could not find container \"1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751\": container with ID starting with 1a32488568bd0c9efc69b79ebbdef6e97ce3b9174c36a0270a04609c88381751 not found: ID does not exist" Oct 30 22:29:19.678895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.678848 4991 scope.go:122] "RemoveContainer" containerID="0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b" Oct 30 22:29:19.679192 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:19.679170 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b\": container with ID starting with 0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b not found: ID does not exist" containerID="0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b" Oct 30 22:29:19.679285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.679198 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b"} err="failed to get container status \"0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b\": rpc error: code = NotFound desc = could not find container \"0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b\": container with ID starting with 0cd676af52d569e85e0a28f95629862f768b64ae638d7a22239aa4499477a96b not found: ID does not exist" Oct 30 22:29:19.761543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.761519 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr"] Oct 30 22:29:19.765888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.765859 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf5019-openshift-default-86d845b745-qtwlr"] Oct 30 22:29:19.776667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.775507 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:29:19.776667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.775561 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:29:19.877639 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:19.877605 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" path="/var/lib/kubelet/pods/0b13dea8-9817-452c-a658-ce355e4b0de6/volumes" Oct 30 22:29:20.595687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.595651 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" event={"ID":"94973040-d19d-4a9a-8b45-b8cdd3bfd176","Type":"ContainerStarted","Data":"792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2"} Oct 30 22:29:20.595869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.595770 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:20.597176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.597152 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" event={"ID":"72224bc6-33bb-43a2-bd01-df0109a97984","Type":"ContainerStarted","Data":"45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89"} Oct 30 22:29:20.597307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.597291 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:20.706632 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.706580 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" podStartSLOduration=1.965269479 podStartE2EDuration="7.706566053s" totalImagesPullingTime="5.741296574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:20.704282516 +0000 UTC m=+2581.003692413" watchObservedRunningTime="2026-10-30 22:29:20.706566053 +0000 UTC m=+2581.005975934" Oct 30 22:29:20.775603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.775562 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:29:20.775744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.775626 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 22:29:20.815849 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:20.815801 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" podStartSLOduration=1.905392189 podStartE2EDuration="7.815788481s" totalImagesPullingTime="5.910396292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:20.81320864 +0000 UTC m=+2581.112618540" watchObservedRunningTime="2026-10-30 22:29:20.815788481 +0000 UTC m=+2581.115198403" Oct 30 22:29:21.797282 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:21.797251 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:21.797792 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:21.797767 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:21.798555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:21.798536 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:29:31.609277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:31.609244 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:29:31.609907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:31.609682 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:29:34.159661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.159628 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn"] Oct 30 22:29:34.160173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.160156 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" Oct 30 22:29:34.160269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.160175 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" Oct 30 22:29:34.160269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.160221 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" Oct 30 22:29:34.160269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.160226 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" Oct 30 22:29:34.160426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.160330 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" Oct 30 22:29:34.165203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.165183 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.215916 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.215829 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c301cf-serving-cert\"" Oct 30 22:29:34.220094 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.220068 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn"] Oct 30 22:29:34.255455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.255424 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-metrics-certs\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.255588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.255497 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252mb\" (UniqueName: \"kubernetes.io/projected/167b67df-0661-4d48-bdf8-40f82638aa13-kube-api-access-252mb\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.255687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.255662 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167b67df-0661-4d48-bdf8-40f82638aa13-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.255827 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.255810 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.357424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.357394 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.357600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.357457 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-metrics-certs\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.357600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.357519 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252mb\" (UniqueName: \"kubernetes.io/projected/167b67df-0661-4d48-bdf8-40f82638aa13-kube-api-access-252mb\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.357600 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:34.357553 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-serving-cert: secret "maas-api-e2e-praxis-b-c301cf-serving-cert" not found Oct 30 22:29:34.357600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.357582 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167b67df-0661-4d48-bdf8-40f82638aa13-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.357829 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:34.357617 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls podName:167b67df-0661-4d48-bdf8-40f82638aa13 nodeName:}" failed. No retries permitted until 2026-10-30 22:29:34.857594987 +0000 UTC m=+2595.157004868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls") pod "maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" (UID: "167b67df-0661-4d48-bdf8-40f82638aa13") : secret "maas-api-e2e-praxis-b-c301cf-serving-cert" not found Oct 30 22:29:34.358385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.358358 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167b67df-0661-4d48-bdf8-40f82638aa13-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.359962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.359940 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-metrics-certs\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.418304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.418229 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252mb\" (UniqueName: \"kubernetes.io/projected/167b67df-0661-4d48-bdf8-40f82638aa13-kube-api-access-252mb\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.863970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.863936 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:34.866460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:34.866425 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:35.080180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:35.080147 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:35.467500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:35.467429 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn"] Oct 30 22:29:35.708004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:35.707970 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" event={"ID":"167b67df-0661-4d48-bdf8-40f82638aa13","Type":"ContainerStarted","Data":"1394cbd873bdd77eeff24ad539bf7730e2e4c97224b2cd40336128baa5cc54d3"} Oct 30 22:29:36.715480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:36.715382 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" event={"ID":"167b67df-0661-4d48-bdf8-40f82638aa13","Type":"ContainerStarted","Data":"bb0bb1346f70b2624e25da725c416b87c51221e1d6856093b227cdfbd3688823"} Oct 30 22:29:36.715480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:36.715453 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:36.828056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:36.827995 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" podStartSLOduration=1.8772421270000001 podStartE2EDuration="2.827982106s" totalImagesPullingTime="950.739979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:36.826109213 +0000 UTC m=+2597.125519098" watchObservedRunningTime="2026-10-30 22:29:36.827982106 +0000 UTC m=+2597.127391985" Oct 30 22:29:42.794388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:42.790544 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:29:47.612378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.612343 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv"] Oct 30 22:29:47.612991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.612975 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b13dea8-9817-452c-a658-ce355e4b0de6" containerName="istio-proxy" Oct 30 22:29:47.616539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.616521 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.643817 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.643792 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.643933 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.643840 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.643981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.643962 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.644035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.643989 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.644114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.644091 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.644172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.644135 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.644231 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.644216 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.644345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.644314 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e648c983-a074-4f69-ad9c-bdf65387e7c1-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.644439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.644424 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmmvq\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-kube-api-access-rmmvq\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.658960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.658936 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv"] Oct 30 22:29:47.662640 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.662623 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.714302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.714281 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv"] Oct 30 22:29:47.745400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745366 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.745554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745418 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.745554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745459 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.745554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745495 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.745554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745523 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.745554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745552 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791caf8-3c35-4a37-9616-3098318e2f72-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.745836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745604 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.745836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745663 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.745836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745720 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.745836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745808 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfd5\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-kube-api-access-shfd5\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.746056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745851 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e648c983-a074-4f69-ad9c-bdf65387e7c1-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.746056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745911 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.746056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745957 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791caf8-3c35-4a37-9616-3098318e2f72-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.746056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.745989 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.746056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.746038 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmmvq\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-kube-api-access-rmmvq\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.746337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.746115 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.746337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.746141 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.746337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.746172 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.746684 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.746659 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.747202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.747172 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.747602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.747576 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.748127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.748103 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.749062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.749037 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e648c983-a074-4f69-ad9c-bdf65387e7c1-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.750122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.750098 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.752958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.752929 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.765830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.765804 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv"] Oct 30 22:29:47.806116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.806080 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmmvq\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-kube-api-access-rmmvq\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.806808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.806781 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.849514 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849473 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849693 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849550 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791caf8-3c35-4a37-9616-3098318e2f72-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849693 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849665 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfd5\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-kube-api-access-shfd5\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849738 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849788 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791caf8-3c35-4a37-9616-3098318e2f72-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849817 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849902 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849935 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.849999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.849975 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.850587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.850476 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.852093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.851952 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.852093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.852030 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.855330 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.855289 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791caf8-3c35-4a37-9616-3098318e2f72-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.855437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.855370 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.861100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.861070 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791caf8-3c35-4a37-9616-3098318e2f72-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.862217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.862188 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.910264 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.910200 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.910448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.910400 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfd5\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-kube-api-access-shfd5\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:47.911443 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:47.911413 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:29:47.930673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.930652 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:47.975932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:47.975899 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:48.364335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.364299 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv"] Oct 30 22:29:48.371919 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:48.371890 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode648c983_a074_4f69_ad9c_bdf65387e7c1.slice/crio-dfe4f238ddb60ac1431d22c4ef109534c33b4d110110d2887b6a0b3fd4c72a38 WatchSource:0}: Error finding container dfe4f238ddb60ac1431d22c4ef109534c33b4d110110d2887b6a0b3fd4c72a38: Status 404 returned error can't find the container with id dfe4f238ddb60ac1431d22c4ef109534c33b4d110110d2887b6a0b3fd4c72a38 Oct 30 22:29:48.374342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.374297 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:48.374438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.374378 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:48.374438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.374406 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:48.418741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.418715 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv"] Oct 30 22:29:48.432465 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:48.432437 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8791caf8_3c35_4a37_9616_3098318e2f72.slice/crio-fdb1c8eff65c418275c434f96046a69228f63c7802ca52e84f6260821189dad5 WatchSource:0}: Error finding container fdb1c8eff65c418275c434f96046a69228f63c7802ca52e84f6260821189dad5: Status 404 returned error can't find the container with id fdb1c8eff65c418275c434f96046a69228f63c7802ca52e84f6260821189dad5 Oct 30 22:29:48.435025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.434991 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:48.435104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.435069 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:48.435144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.435112 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:48.857056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.857004 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" event={"ID":"8791caf8-3c35-4a37-9616-3098318e2f72","Type":"ContainerStarted","Data":"3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c"} Oct 30 22:29:48.857619 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.857084 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" event={"ID":"8791caf8-3c35-4a37-9616-3098318e2f72","Type":"ContainerStarted","Data":"fdb1c8eff65c418275c434f96046a69228f63c7802ca52e84f6260821189dad5"} Oct 30 22:29:48.859395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.859351 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" event={"ID":"e648c983-a074-4f69-ad9c-bdf65387e7c1","Type":"ContainerStarted","Data":"b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4"} Oct 30 22:29:48.859505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.859404 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" event={"ID":"e648c983-a074-4f69-ad9c-bdf65387e7c1","Type":"ContainerStarted","Data":"dfe4f238ddb60ac1431d22c4ef109534c33b4d110110d2887b6a0b3fd4c72a38"} Oct 30 22:29:48.931819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.931773 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:48.978346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.978115 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:48.994340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:48.993681 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podStartSLOduration=1.9936632319999998 podStartE2EDuration="1.993663232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:48.988543352 +0000 UTC m=+2609.287971392" watchObservedRunningTime="2026-10-30 22:29:48.993663232 +0000 UTC m=+2609.293073113" Oct 30 22:29:49.112133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:49.111925 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podStartSLOduration=2.1119072819999998 podStartE2EDuration="2.111907282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:49.110916027 +0000 UTC m=+2609.410325907" watchObservedRunningTime="2026-10-30 22:29:49.111907282 +0000 UTC m=+2609.411317164" Oct 30 22:29:49.940344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:49.937437 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:29:49.940344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:49.937512 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:29:49.977862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:49.977823 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:29:49.978040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:49.977879 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:29:50.945273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:50.945236 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:29:50.945875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:50.945843 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:29:50.976846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:50.976812 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:29:50.976981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:50.976897 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:29:51.940343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:51.939743 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:29:51.940343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:51.939809 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:29:51.977217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:51.977179 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:29:51.977635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:51.977244 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:29:52.939346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:52.932645 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:29:52.939346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:52.932702 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:29:52.977123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:52.977080 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:29:52.977292 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:52.977148 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:29:53.249344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.245802 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt"] Oct 30 22:29:53.280602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.279632 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.293009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.292735 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt"] Oct 30 22:29:53.342694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.342649 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-28f124-serving-cert\"" Oct 30 22:29:53.368395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.360043 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d7f3e8-b372-4e10-96ca-9099a500c474-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.368395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.360136 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmvp\" (UniqueName: \"kubernetes.io/projected/74d7f3e8-b372-4e10-96ca-9099a500c474-kube-api-access-flmvp\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.368395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.360213 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.368395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.360312 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-metrics-certs\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.471349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.468186 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.471349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.468304 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-metrics-certs\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.471349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.468428 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d7f3e8-b372-4e10-96ca-9099a500c474-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.471349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.468499 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmvp\" (UniqueName: \"kubernetes.io/projected/74d7f3e8-b372-4e10-96ca-9099a500c474-kube-api-access-flmvp\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.471349 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:53.468947 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-serving-cert: secret "maas-api-e2e-praxis-b-28f124-serving-cert" not found Oct 30 22:29:53.471349 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:29:53.469012 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls podName:74d7f3e8-b372-4e10-96ca-9099a500c474 nodeName:}" failed. No retries permitted until 2026-10-30 22:29:53.968990309 +0000 UTC m=+2614.268400182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls") pod "maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" (UID: "74d7f3e8-b372-4e10-96ca-9099a500c474") : secret "maas-api-e2e-praxis-b-28f124-serving-cert" not found Oct 30 22:29:53.471349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.470356 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d7f3e8-b372-4e10-96ca-9099a500c474-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.556760 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.556668 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-metrics-certs\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.569617 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.561414 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmvp\" (UniqueName: \"kubernetes.io/projected/74d7f3e8-b372-4e10-96ca-9099a500c474-kube-api-access-flmvp\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.944394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.943754 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:29:53.944394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.943816 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:29:53.980347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.979775 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:29:53.980347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.979841 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:29:53.985668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.985636 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:53.988165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:53.988144 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls\") pod \"maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:54.197513 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:54.197437 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:54.639172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:54.639135 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt"] Oct 30 22:29:54.645949 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:54.645921 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d7f3e8_b372_4e10_96ca_9099a500c474.slice/crio-be2ca8fd8d4fa12e8921ae218e7eddf7d207231938d4d4943faa84f5524d291f WatchSource:0}: Error finding container be2ca8fd8d4fa12e8921ae218e7eddf7d207231938d4d4943faa84f5524d291f: Status 404 returned error can't find the container with id be2ca8fd8d4fa12e8921ae218e7eddf7d207231938d4d4943faa84f5524d291f Oct 30 22:29:54.911359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:54.911246 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" event={"ID":"74d7f3e8-b372-4e10-96ca-9099a500c474","Type":"ContainerStarted","Data":"be2ca8fd8d4fa12e8921ae218e7eddf7d207231938d4d4943faa84f5524d291f"} Oct 30 22:29:54.931680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:54.931650 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:29:54.931794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:54.931707 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:29:54.977027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:54.976992 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:29:54.977187 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:54.977046 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:29:55.922909 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:55.922812 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" event={"ID":"74d7f3e8-b372-4e10-96ca-9099a500c474","Type":"ContainerStarted","Data":"bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b"} Oct 30 22:29:55.923801 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:55.923777 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:29:55.986374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:55.986345 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:55.986773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:55.986725 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:55.987571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:55.987552 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:29:56.025647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.025619 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4"] Oct 30 22:29:56.041133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.041097 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.073609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.073538 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" podStartSLOduration=2.177465252 podStartE2EDuration="3.073519121s" totalImagesPullingTime="896.053869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:56.044358869 +0000 UTC m=+2616.343768777" watchObservedRunningTime="2026-10-30 22:29:56.073519121 +0000 UTC m=+2616.372929002" Oct 30 22:29:56.090672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.090625 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:56.091507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.091483 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:56.095453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.094163 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4"] Oct 30 22:29:56.097724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.097698 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-9717a8-openshift-default-dockercfg-n4n7b\"" Oct 30 22:29:56.098892 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.098873 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:29:56.130184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.130147 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-data\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.130470 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.130441 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-token\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.130658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.130638 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc9w\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-kube-api-access-wnc9w\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.130824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.130807 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-credential-socket\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.130958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.130943 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-socket\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.131143 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.131123 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-certs\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.131250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.131230 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istiod-ca-cert\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.131462 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.131438 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-podinfo\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.131621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.131603 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-envoy\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234065 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-data\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234149 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-token\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234189 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc9w\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-kube-api-access-wnc9w\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234227 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-credential-socket\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234273 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-socket\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234368 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-certs\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234496 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istiod-ca-cert\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234529 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-podinfo\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234577 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-envoy\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.234989 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.234966 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-credential-socket\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.235159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.235141 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-data\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.235437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.235406 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-certs\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.235539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.235470 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-socket\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.236067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.236046 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istiod-ca-cert\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.238185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.238125 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-podinfo\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.238582 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.238536 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-envoy\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.296533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.296497 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc9w\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-kube-api-access-wnc9w\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.298140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.298083 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-token\") pod \"e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.376289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.376259 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:56.882804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.882726 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4"] Oct 30 22:29:56.889389 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:29:56.889358 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4cc59e_42c4_4e5b_b996_2b4cf886879b.slice/crio-c684a8c116a820ba43aaa657c687499129dd089f01952c7852162988b739de04 WatchSource:0}: Error finding container c684a8c116a820ba43aaa657c687499129dd089f01952c7852162988b739de04: Status 404 returned error can't find the container with id c684a8c116a820ba43aaa657c687499129dd089f01952c7852162988b739de04 Oct 30 22:29:56.891986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.891954 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:56.892085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.892021 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:56.892085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.892051 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:29:56.931150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:56.931122 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" event={"ID":"8b4cc59e-42c4-4e5b-b996-2b4cf886879b","Type":"ContainerStarted","Data":"c684a8c116a820ba43aaa657c687499129dd089f01952c7852162988b739de04"} Oct 30 22:29:57.940481 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:57.940440 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" event={"ID":"8b4cc59e-42c4-4e5b-b996-2b4cf886879b","Type":"ContainerStarted","Data":"baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419"} Oct 30 22:29:58.128871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:58.128823 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podStartSLOduration=3.128810578 podStartE2EDuration="3.128810578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:58.101786097 +0000 UTC m=+2618.401195998" watchObservedRunningTime="2026-10-30 22:29:58.128810578 +0000 UTC m=+2618.428220457" Oct 30 22:29:58.376476 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:58.376443 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:58.382028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:58.382004 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:58.946925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:58.946894 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:29:58.947831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:29:58.947813 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:30:02.947480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:02.947446 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:30:08.689899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.689860 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c"] Oct 30 22:30:08.716594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.716563 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.904949 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-podinfo\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905004 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648tx\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-kube-api-access-648tx\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905053 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-envoy\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905145 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-token\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905230 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-certs\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905259 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-data\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905306 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-credential-socket\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905357 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-socket\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.905427 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istiod-ca-cert\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:08.945418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.938038 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f39eb7-openshift-default-dockercfg-4npcr\"" Oct 30 22:30:08.964425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:08.964380 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c"] Oct 30 22:30:09.084353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.084046 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-certs\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.084353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.084095 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-data\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.084353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.084167 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-credential-socket\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.084353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.084199 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-socket\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.084353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.084297 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istiod-ca-cert\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.093446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.085234 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-podinfo\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.093446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.085270 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648tx\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-kube-api-access-648tx\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.093446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.085337 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-envoy\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.093446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.085460 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-token\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.093446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.086599 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istiod-ca-cert\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.093446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.087059 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-certs\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.093446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.087279 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-data\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.102425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.094521 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-socket\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.102425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.094744 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-credential-socket\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.230563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.200125 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-token\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.232200 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.232133 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-podinfo\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.241390 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.241364 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-648tx\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-kube-api-access-648tx\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.265432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.241771 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-envoy\") pod \"e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:09.549399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.532546 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:10.001344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:09.998399 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 22:30:10.084342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:10.083469 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:10.371357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:10.346404 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:10.697436 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:10.697344 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c"] Oct 30 22:30:11.021453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.004672 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:30:11.021453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.004740 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:11.078643 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.042303 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 22:30:11.078643 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.042586 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:30:11.093115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.090433 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:11.093115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.091126 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:30:11.183257 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.183208 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:11.183545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.183522 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:11.183765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.183723 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:11.219239 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.219214 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:30:11.230439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.230414 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:30:11.508729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.508689 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" event={"ID":"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7","Type":"ContainerStarted","Data":"dd374013d2235ce2db4b4ac966e5061743fd9a89bdbdc6accaf67378f2a4290f"} Oct 30 22:30:11.536473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.536439 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:11.699730 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:11.699617 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podStartSLOduration=3.699596782 podStartE2EDuration="3.699596782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:11.693826823 +0000 UTC m=+2631.993236706" watchObservedRunningTime="2026-10-30 22:30:11.699596782 +0000 UTC m=+2631.999006665" Oct 30 22:30:12.516795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:12.516742 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" event={"ID":"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7","Type":"ContainerStarted","Data":"88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc"} Oct 30 22:30:12.556344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:12.555832 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:13.542345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.538205 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:13.550952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.550922 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:30:13.900504 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.900432 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt"] Oct 30 22:30:13.904937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.904919 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.998814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.998778 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt"] Oct 30 22:30:13.999086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999063 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999096 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999211 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02424f4-1f21-461e-9f87-dd21407f5292-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999292 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999372 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999516 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999580 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999619 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02424f4-1f21-461e-9f87-dd21407f5292-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:13.999675 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:13.999652 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84g7j\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-kube-api-access-84g7j\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.100842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.100809 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02424f4-1f21-461e-9f87-dd21407f5292-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.100860 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.100983 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101077 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101098 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101121 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02424f4-1f21-461e-9f87-dd21407f5292-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101147 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84g7j\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-kube-api-access-84g7j\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101231 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101236 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101250 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101308 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101645 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101496 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101759 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02424f4-1f21-461e-9f87-dd21407f5292-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.101861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.101833 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.103498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.103475 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.103900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.103880 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02424f4-1f21-461e-9f87-dd21407f5292-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.172009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.171925 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84g7j\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-kube-api-access-84g7j\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.173628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.173603 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.218369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.218343 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:14.713588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.713554 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt"] Oct 30 22:30:14.723922 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:30:14.723870 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02424f4_1f21_461e_9f87_dd21407f5292.slice/crio-a68fddc74ff1be9fc340d9d9bf210335a232275d43bdb850ba9e9e66791e057b WatchSource:0}: Error finding container a68fddc74ff1be9fc340d9d9bf210335a232275d43bdb850ba9e9e66791e057b: Status 404 returned error can't find the container with id a68fddc74ff1be9fc340d9d9bf210335a232275d43bdb850ba9e9e66791e057b Oct 30 22:30:14.730926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.730886 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:14.731123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.731103 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:14.731253 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:14.731235 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:15.563720 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:15.563665 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" event={"ID":"c02424f4-1f21-461e-9f87-dd21407f5292","Type":"ContainerStarted","Data":"19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1"} Oct 30 22:30:15.564300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:15.564258 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" event={"ID":"c02424f4-1f21-461e-9f87-dd21407f5292","Type":"ContainerStarted","Data":"a68fddc74ff1be9fc340d9d9bf210335a232275d43bdb850ba9e9e66791e057b"} Oct 30 22:30:15.682144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:15.682086 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podStartSLOduration=2.682072529 podStartE2EDuration="2.682072529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:15.677498731 +0000 UTC m=+2635.976908647" watchObservedRunningTime="2026-10-30 22:30:15.682072529 +0000 UTC m=+2635.981482406" Oct 30 22:30:16.218533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:16.218485 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:16.229522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:16.229486 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:30:16.243492 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:16.243448 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 22:30:17.219164 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:17.219123 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:30:17.219656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:17.219185 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 22:30:17.914770 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:30:17.914729 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:30:18.225483 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:18.223970 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:30:18.225483 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:18.224027 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 22:30:19.219806 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:19.219356 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:30:19.219806 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:19.219413 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 22:30:20.223349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:20.220071 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:30:20.223349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:20.220130 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 22:30:21.221072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.220911 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:30:21.221415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.221385 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 22:30:21.630401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.630358 4991 generic.go:356] "Generic (PLEG): container finished" podID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerID="7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27" exitCode=0 Oct 30 22:30:21.630901 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.630410 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" event={"ID":"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c","Type":"ContainerDied","Data":"7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27"} Oct 30 22:30:21.630965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.630943 4991 scope.go:122] "RemoveContainer" containerID="7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27" Oct 30 22:30:21.632529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.632503 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:21.632626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.632571 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:21.632626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:21.632602 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:22.423448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:22.423411 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:30:22.589087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:22.589054 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:22.589541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:22.589516 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:22.590268 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:22.590249 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:30:22.656356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:22.654471 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" event={"ID":"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c","Type":"ContainerStarted","Data":"c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1"} Oct 30 22:30:23.362039 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:23.362008 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:30:23.363379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:23.363354 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:30:23.363503 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:23.363402 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:30:24.362315 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:24.362281 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:30:24.362740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:24.362357 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:30:25.362768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:25.362731 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:30:25.363349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:25.363303 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:30:26.361758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:26.361720 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:30:26.361915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:26.361794 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:30:27.361925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:27.361890 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:30:27.362418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:27.361953 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:30:28.367681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:28.367645 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:30:28.368126 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:28.367979 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:30:28.369078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:28.369057 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:30:29.026854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.026820 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch"] Oct 30 22:30:29.030717 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.030696 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.118368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.118314 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch"] Oct 30 22:30:29.189991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.189957 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190012 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190100 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190136 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190168 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190249 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190302 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190465 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190430 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.190540 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.190523 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxs2\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-kube-api-access-5xxs2\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.291744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291665 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.291744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291722 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.291744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291745 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291768 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291796 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291828 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291886 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.291922 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.292003 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxs2\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-kube-api-access-5xxs2\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292491 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.292351 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.292540 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292684 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.292662 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.292683 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.292783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.292762 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.294219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.294188 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.294556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.294538 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.350696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.350669 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.350816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.350740 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxs2\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-kube-api-access-5xxs2\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:29.643521 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:29.643438 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:30.080016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:30.079979 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch"] Oct 30 22:30:30.087071 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:30:30.087040 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde3b88b_30ce_4dc2_9a43_00145f5d964a.slice/crio-e3c1d7750f5ca1594fc19ead7a7eeb704563ab4febdce77d210a6fa63a153d7c WatchSource:0}: Error finding container e3c1d7750f5ca1594fc19ead7a7eeb704563ab4febdce77d210a6fa63a153d7c: Status 404 returned error can't find the container with id e3c1d7750f5ca1594fc19ead7a7eeb704563ab4febdce77d210a6fa63a153d7c Oct 30 22:30:30.089511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:30.089480 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:30.089612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:30.089547 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:30.089612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:30.089573 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:30.717305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:30.717261 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" event={"ID":"cde3b88b-30ce-4dc2-9a43-00145f5d964a","Type":"ContainerStarted","Data":"9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835"} Oct 30 22:30:30.717727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:30.717382 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" event={"ID":"cde3b88b-30ce-4dc2-9a43-00145f5d964a","Type":"ContainerStarted","Data":"e3c1d7750f5ca1594fc19ead7a7eeb704563ab4febdce77d210a6fa63a153d7c"} Oct 30 22:30:30.834932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:30.834863 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podStartSLOduration=2.834844705 podStartE2EDuration="2.834844705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:30.832122357 +0000 UTC m=+2651.131532238" watchObservedRunningTime="2026-10-30 22:30:30.834844705 +0000 UTC m=+2651.134254587" Oct 30 22:30:31.651340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:31.648868 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:31.651340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:31.649027 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:30:31.651340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:31.649073 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 22:30:32.650158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:32.648425 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:30:32.650158 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:32.648498 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 22:30:33.648343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:33.647709 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:30:33.648343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:33.647774 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 22:30:34.648680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:34.648642 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:30:34.649176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:34.648717 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 22:30:35.650347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:35.644629 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:30:35.650347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:35.644702 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 22:30:36.405627 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.405595 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2"] Oct 30 22:30:36.410824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.410800 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.475679 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.474698 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-fc6466-openshift-default-dockercfg-7d6qb\"" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584279 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-certs\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584354 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-podinfo\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584423 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-token\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584462 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-socket\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584489 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfzk\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-kube-api-access-8mfzk\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584583 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istiod-ca-cert\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584622 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-credential-socket\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584674 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-envoy\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.587345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.584744 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-data\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.607110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.607065 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2"] Oct 30 22:30:36.644448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.644415 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:30:36.644687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.644664 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 22:30:36.686251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686166 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-envoy\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686225 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-data\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686304 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-certs\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686340 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-podinfo\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686388 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-token\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686412 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-socket\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686441 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfzk\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-kube-api-access-8mfzk\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686528 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istiod-ca-cert\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686564 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-credential-socket\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.686854 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686826 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-data\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.687296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.686861 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-certs\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.687296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.687091 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-socket\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.687442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.687419 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istiod-ca-cert\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.687537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.687502 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-credential-socket\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.688944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.688918 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-envoy\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.689863 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.689840 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-podinfo\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.746439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.746377 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-token\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.753351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.753263 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfzk\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-kube-api-access-8mfzk\") pod \"e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:36.784222 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:36.784176 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:37.226793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:37.226756 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2"] Oct 30 22:30:37.233075 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:30:37.233045 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4e09b8_0c01_42ab_a5b7_ff487d4c0aa6.slice/crio-573db9f45cd5ffae326e085c36e6808d0864f7430ea8484ae382a351abd1c9f3 WatchSource:0}: Error finding container 573db9f45cd5ffae326e085c36e6808d0864f7430ea8484ae382a351abd1c9f3: Status 404 returned error can't find the container with id 573db9f45cd5ffae326e085c36e6808d0864f7430ea8484ae382a351abd1c9f3 Oct 30 22:30:37.235688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:37.235654 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:37.235805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:37.235719 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:37.235805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:37.235749 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:30:37.816339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:37.810918 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" event={"ID":"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6","Type":"ContainerStarted","Data":"69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54"} Oct 30 22:30:37.816339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:37.810986 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" event={"ID":"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6","Type":"ContainerStarted","Data":"573db9f45cd5ffae326e085c36e6808d0864f7430ea8484ae382a351abd1c9f3"} Oct 30 22:30:37.952343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:37.948723 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podStartSLOduration=1.9487040279999999 podStartE2EDuration="1.948704028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:37.942597831 +0000 UTC m=+2658.242007707" watchObservedRunningTime="2026-10-30 22:30:37.948704028 +0000 UTC m=+2658.248113908" Oct 30 22:30:38.235483 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:38.235445 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:38.236864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:38.236839 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:38.240515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:38.240461 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:30:38.899344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:38.882593 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:38.969345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:38.911123 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:39.033340 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:39.032794 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:39.054764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:39.054738 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:30:39.490570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:39.490230 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:40.231465 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:40.231403 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:40.538548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:40.538452 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" podUID="227fcbbf-d027-4db6-b8ec-03659183f431" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:40.539412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:40.539384 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 22:30:40.891395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:40.890877 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t" Oct 30 22:30:47.911939 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:30:47.911901 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:30:56.842380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.842344 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft"] Oct 30 22:30:56.847466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.847445 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:56.943661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.943628 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1875b4\"/\"openshift-service-ca.crt\"" Oct 30 22:30:56.943809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.943691 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1875b4\"/\"praxis-route-1875b4-kserve-self-signed-certs\"" Oct 30 22:30:56.943887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.943832 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1875b4\"/\"default-dockercfg-xnmls\"" Oct 30 22:30:56.943937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.943883 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1875b4\"/\"kube-root-ca.crt\"" Oct 30 22:30:56.947329 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.947298 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft"] Oct 30 22:30:56.978521 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.978495 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-tmp-dir\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:56.978652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.978527 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-dshm\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:56.978652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.978594 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-home\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:56.978652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.978647 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjmp\" (UniqueName: \"kubernetes.io/projected/8129b506-3ca3-489f-8b34-c9bfd438b93e-kube-api-access-5cjmp\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:56.978859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.978838 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-model-cache\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:56.978922 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:56.978899 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129b506-3ca3-489f-8b34-c9bfd438b93e-tls-certs\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.079843 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.079813 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-tmp-dir\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080054 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.079860 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-dshm\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080054 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.079903 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-home\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080054 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.079943 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjmp\" (UniqueName: \"kubernetes.io/projected/8129b506-3ca3-489f-8b34-c9bfd438b93e-kube-api-access-5cjmp\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.080065 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-model-cache\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.080119 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129b506-3ca3-489f-8b34-c9bfd438b93e-tls-certs\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.080228 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-tmp-dir\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.080350 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-home\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.080475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.080456 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-model-cache\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.082376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.082343 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-dshm\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.082599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.082573 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129b506-3ca3-489f-8b34-c9bfd438b93e-tls-certs\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.138181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.138119 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjmp\" (UniqueName: \"kubernetes.io/projected/8129b506-3ca3-489f-8b34-c9bfd438b93e-kube-api-access-5cjmp\") pod \"praxis-route-1875b4-kserve-5fc4b79f4c-fjpft\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.160941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.160920 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:30:57.542646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:57.542616 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft"] Oct 30 22:30:57.548533 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:30:57.548507 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8129b506_3ca3_489f_8b34_c9bfd438b93e.slice/crio-0cc8c12e212b78c5bf9fe050c3ea2de5ac502a87811e7870fad71e0ef14665bd WatchSource:0}: Error finding container 0cc8c12e212b78c5bf9fe050c3ea2de5ac502a87811e7870fad71e0ef14665bd: Status 404 returned error can't find the container with id 0cc8c12e212b78c5bf9fe050c3ea2de5ac502a87811e7870fad71e0ef14665bd Oct 30 22:30:58.234335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:58.234275 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" event={"ID":"8129b506-3ca3-489f-8b34-c9bfd438b93e","Type":"ContainerStarted","Data":"cf0ba964f8d3156d65db39da9043e063f66f1fe2e5071f27a92b94caf7c09135"} Oct 30 22:30:58.234747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:58.234376 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" event={"ID":"8129b506-3ca3-489f-8b34-c9bfd438b93e","Type":"ContainerStarted","Data":"0cc8c12e212b78c5bf9fe050c3ea2de5ac502a87811e7870fad71e0ef14665bd"} Oct 30 22:30:58.350930 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:58.350870 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" podStartSLOduration=2.350858185 podStartE2EDuration="2.350858185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:58.347910895 +0000 UTC m=+2678.647320778" watchObservedRunningTime="2026-10-30 22:30:58.350858185 +0000 UTC m=+2678.650268066" Oct 30 22:30:59.161392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:30:59.161360 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:31:07.161923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:07.161881 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:31:07.174102 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:07.174076 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:31:07.337161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:07.337127 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:31:11.854486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:11.854453 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g"] Oct 30 22:31:11.855054 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:11.854713 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" containerID="cri-o://be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088" gracePeriod=30 Oct 30 22:31:17.151967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.151937 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:31:17.237495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237458 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-certs\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.237695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237536 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-token\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-token\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.237695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237591 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-podinfo\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.237805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237699 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d199c57d-49ec-4b6b-8cee-6511796ee6de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d199c57d-49ec-4b6b-8cee-6511796ee6de-istiod-ca-cert\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.237805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237758 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-credential-socket\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.237805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237766 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-certs" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:17.237972 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237824 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-kube-api-access-v5f6m\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-kube-api-access-v5f6m\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.237972 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237954 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-data\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.238090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.237990 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-envoy\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.238090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.238037 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-socket\") pod \"d199c57d-49ec-4b6b-8cee-6511796ee6de\" (UID: \"d199c57d-49ec-4b6b-8cee-6511796ee6de\") " Oct 30 22:31:17.238834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.238613 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.238834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.238678 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-credential-socket" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:17.238834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.238740 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-data" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:17.238834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.238815 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-socket" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:17.239072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.238862 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d199c57d-49ec-4b6b-8cee-6511796ee6de-istiod-ca-cert" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:17.241293 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.241265 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-token" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:17.241293 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.241286 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-kube-api-access-v5f6m" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "kube-api-access-v5f6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:17.241490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.241462 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-envoy" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:17.241555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.241489 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-podinfo" pod "d199c57d-49ec-4b6b-8cee-6511796ee6de" (UID: "d199c57d-49ec-4b6b-8cee-6511796ee6de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:17.340413 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340362 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.340413 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340399 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.340413 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340415 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d199c57d-49ec-4b6b-8cee-6511796ee6de-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.340764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340431 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.340764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340447 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5f6m\" (UniqueName: \"kubernetes.io/projected/d199c57d-49ec-4b6b-8cee-6511796ee6de-kube-api-access-v5f6m\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.340764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340462 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.340764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340475 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.340764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.340489 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d199c57d-49ec-4b6b-8cee-6511796ee6de-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:17.414728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.414643 4991 generic.go:356] "Generic (PLEG): container finished" podID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerID="be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088" exitCode=0 Oct 30 22:31:17.414887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.414723 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" Oct 30 22:31:17.414887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.414731 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" event={"ID":"d199c57d-49ec-4b6b-8cee-6511796ee6de","Type":"ContainerDied","Data":"be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088"} Oct 30 22:31:17.414887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.414787 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g" event={"ID":"d199c57d-49ec-4b6b-8cee-6511796ee6de","Type":"ContainerDied","Data":"78312b5781a18baed0e40ad67d7b24ec8a4490282b158e23f594637d0b454a08"} Oct 30 22:31:17.414887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.414813 4991 scope.go:122] "RemoveContainer" containerID="be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088" Oct 30 22:31:17.428091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.428074 4991 scope.go:122] "RemoveContainer" containerID="a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927" Oct 30 22:31:17.437747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.437727 4991 scope.go:122] "RemoveContainer" containerID="be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088" Oct 30 22:31:17.438032 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:31:17.438009 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088\": container with ID starting with be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088 not found: ID does not exist" containerID="be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088" Oct 30 22:31:17.438126 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.438041 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088"} err="failed to get container status \"be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088\": rpc error: code = NotFound desc = could not find container \"be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088\": container with ID starting with be1c70985bfa272a59bd16918c60566f79d69e4c5ae34176fa98cd6327d84088 not found: ID does not exist" Oct 30 22:31:17.438126 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.438059 4991 scope.go:122] "RemoveContainer" containerID="a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927" Oct 30 22:31:17.438348 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:31:17.438315 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927\": container with ID starting with a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927 not found: ID does not exist" containerID="a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927" Oct 30 22:31:17.438408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.438355 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927"} err="failed to get container status \"a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927\": rpc error: code = NotFound desc = could not find container \"a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927\": container with ID starting with a13f2412092bbce4fa51c230e1b316b0a5ea1c4d322300f21c8da4bd52e43927 not found: ID does not exist" Oct 30 22:31:17.575226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.575193 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g"] Oct 30 22:31:17.580561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.580534 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-459aa8-openshift-default-6fcfd7f949-r555g"] Oct 30 22:31:17.877664 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:17.877624 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" path="/var/lib/kubelet/pods/d199c57d-49ec-4b6b-8cee-6511796ee6de/volumes" Oct 30 22:31:17.911899 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:31:17.911849 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:31:18.424272 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:18.424235 4991 generic.go:356] "Generic (PLEG): container finished" podID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerID="baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419" exitCode=0 Oct 30 22:31:18.424762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:18.424290 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" event={"ID":"8b4cc59e-42c4-4e5b-b996-2b4cf886879b","Type":"ContainerDied","Data":"baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419"} Oct 30 22:31:18.424762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:18.424752 4991 scope.go:122] "RemoveContainer" containerID="baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419" Oct 30 22:31:18.426258 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:18.426224 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:18.426385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:18.426296 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:18.426385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:18.426347 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:19.449877 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:19.449841 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" event={"ID":"8b4cc59e-42c4-4e5b-b996-2b4cf886879b","Type":"ContainerStarted","Data":"7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8"} Oct 30 22:31:20.365209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:20.365180 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:31:20.376464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:20.376437 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:31:20.376692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:20.376664 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:31:20.376803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:20.376719 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 22:31:20.380827 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:20.380810 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:31:21.377571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:21.377534 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:31:21.377985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:21.377594 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 22:31:22.377061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:22.377022 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:31:22.377251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:22.377075 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 22:31:23.377069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:23.377028 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:31:23.377513 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:23.377086 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 22:31:24.377221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:24.377188 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:31:24.377647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:24.377245 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 22:31:25.382834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:25.382802 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:31:25.383364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:25.383343 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:31:25.384022 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:25.384005 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:31:44.579472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:44.579429 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:31:44.580050 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:44.579489 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:31:46.651927 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:46.651882 4991 generic.go:356] "Generic (PLEG): container finished" podID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerID="88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc" exitCode=0 Oct 30 22:31:46.652485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:46.651960 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" event={"ID":"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7","Type":"ContainerDied","Data":"88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc"} Oct 30 22:31:46.652485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:46.652476 4991 scope.go:122] "RemoveContainer" containerID="88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc" Oct 30 22:31:46.654091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:46.654064 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:46.654195 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:46.654130 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:46.654195 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:46.654179 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:31:47.663110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:47.663072 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" event={"ID":"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7","Type":"ContainerStarted","Data":"cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505"} Oct 30 22:31:47.911344 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:31:47.911297 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:31:48.541354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:48.540676 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:31:48.541354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:48.540800 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:31:48.541354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:48.540843 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 22:31:49.534342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:49.533666 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:31:49.534342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:49.533727 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 22:31:50.540350 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:50.533672 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:31:50.540350 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:50.533731 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 22:31:51.533758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:51.533719 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:31:51.534153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:51.534111 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 22:31:52.536344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:52.533695 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:31:52.536344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:52.533755 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 22:31:53.538863 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:53.538831 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:31:53.539370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:53.539181 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:31:53.539996 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:53.539980 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:31:59.648773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.648737 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs"] Oct 30 22:31:59.649407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.649389 4991 cpu_manager.go:414] "containerMap: removing container" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" Oct 30 22:31:59.649407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.649406 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" Oct 30 22:31:59.649541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.649532 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" Oct 30 22:31:59.652042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.652023 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.699160 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.699117 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-socket\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.700644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.700616 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istiod-ca-cert\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.700888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.700869 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-token\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.707846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.707823 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-envoy\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.708027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.708011 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-podinfo\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.708182 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.708166 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-data\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.708309 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.708292 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-certs\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.708467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.708450 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-credential-socket\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.708581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.708565 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48nr\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-kube-api-access-r48nr\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.829394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.822652 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-68f80c-openshift-default-dockercfg-jw5t9\"" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830441 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-socket\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830589 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istiod-ca-cert\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830673 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-token\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830749 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-envoy\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830776 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-podinfo\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830807 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-data\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830847 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-certs\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830886 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-credential-socket\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.832172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.830917 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48nr\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-kube-api-access-r48nr\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.858165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.858123 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istiod-ca-cert\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.858587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.858551 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-data\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.858894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.858846 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-certs\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.859130 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.859109 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-credential-socket\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.863078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.861592 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-podinfo\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.900108 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.900033 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48nr\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-kube-api-access-r48nr\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.939737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.937526 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-socket\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.986589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.986441 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-token\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:31:59.990304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.990270 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs"] Oct 30 22:31:59.991598 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:31:59.991572 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-envoy\") pod \"e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:32:00.267341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:00.267122 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:32:00.763530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:00.763257 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs"] Oct 30 22:32:00.766603 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:32:00.766570 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39de1aba_6701_42bf_bcfd_d2327f97f5a5.slice/crio-f224eb4040f68f646368e400507b4e59680311d784ad3b274d5b7b8519f55259 WatchSource:0}: Error finding container f224eb4040f68f646368e400507b4e59680311d784ad3b274d5b7b8519f55259: Status 404 returned error can't find the container with id f224eb4040f68f646368e400507b4e59680311d784ad3b274d5b7b8519f55259 Oct 30 22:32:00.769210 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:00.769178 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:00.769283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:00.769248 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:00.769283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:00.769277 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:00.915733 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:00.915689 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" event={"ID":"39de1aba-6701-42bf-bcfd-d2327f97f5a5","Type":"ContainerStarted","Data":"7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83"} Oct 30 22:32:00.915904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:00.915753 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" event={"ID":"39de1aba-6701-42bf-bcfd-d2327f97f5a5","Type":"ContainerStarted","Data":"f224eb4040f68f646368e400507b4e59680311d784ad3b274d5b7b8519f55259"} Oct 30 22:32:01.056401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:01.056003 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podStartSLOduration=2.055981809 podStartE2EDuration="2.055981809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:01.043598306 +0000 UTC m=+2741.343008213" watchObservedRunningTime="2026-10-30 22:32:01.055981809 +0000 UTC m=+2741.355391689" Oct 30 22:32:01.270395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:01.269154 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:32:01.270395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:01.269556 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:32:01.270395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:01.269601 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:32:02.275576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:02.275541 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:32:02.956964 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:02.956861 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:32:02.957768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:02.957748 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:32:06.987981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:06.987950 4991 generic.go:356] "Generic (PLEG): container finished" podID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerID="69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54" exitCode=0 Oct 30 22:32:06.988404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:06.988025 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" event={"ID":"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6","Type":"ContainerDied","Data":"69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54"} Oct 30 22:32:06.988491 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:06.988471 4991 scope.go:122] "RemoveContainer" containerID="69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54" Oct 30 22:32:06.989946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:06.989921 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:06.990025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:06.989993 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:06.990063 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:06.990024 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:07.996747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:07.996704 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" event={"ID":"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6","Type":"ContainerStarted","Data":"70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64"} Oct 30 22:32:08.784539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:08.784508 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:32:08.785456 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:08.785414 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 22:32:08.785596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:08.785487 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 22:32:09.045976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:09.045881 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:32:09.785057 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:09.785023 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 22:32:09.785234 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:09.785077 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 22:32:10.785611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:10.785574 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 22:32:10.786010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:10.785626 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 22:32:11.785167 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:11.785128 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 22:32:11.785376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:11.785193 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 22:32:12.784655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:12.784621 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 22:32:12.785064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:12.784679 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 22:32:13.158165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.158089 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq"] Oct 30 22:32:13.158764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.158746 4991 cpu_manager.go:414] "containerMap: removing container" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" Oct 30 22:32:13.158812 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.158765 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" Oct 30 22:32:13.158895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.158886 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d199c57d-49ec-4b6b-8cee-6511796ee6de" containerName="istio-proxy" Oct 30 22:32:13.161487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.161467 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.216365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.216305 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-socket\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.216665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.216634 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-envoy\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.216810 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.216791 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kj5f\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-kube-api-access-7kj5f\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.216980 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.216959 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istiod-ca-cert\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.217163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.217140 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-credential-socket\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.217450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.217406 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-podinfo\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.217773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.217745 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-data\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.217910 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.217892 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-certs\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.230339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.218031 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-token\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.231136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.218437 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-37e79b-openshift-default-dockercfg-989jt\"" Oct 30 22:32:13.341354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.333204 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-socket\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.341354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.333278 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-envoy\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.341354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.333302 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kj5f\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-kube-api-access-7kj5f\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.341354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.333352 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istiod-ca-cert\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.341354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.333389 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-credential-socket\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.341354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.333476 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-podinfo\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.341354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.335197 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-socket\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.365543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.363528 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-data\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.365543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.363587 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-certs\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.365543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.363618 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-token\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.404134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.404101 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kj5f\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-kube-api-access-7kj5f\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.405822 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.405795 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq"] Oct 30 22:32:13.415398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.414651 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-data\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.419259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.415975 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-certs\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.420312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.420241 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istiod-ca-cert\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.420683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.420638 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-credential-socket\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.442339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.437283 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-token\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.442339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.437601 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-envoy\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.454555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.454444 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-podinfo\") pod \"e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.474903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.474877 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:13.789479 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.789451 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:32:13.790486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.790465 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:32:13.915947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.915912 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq"] Oct 30 22:32:13.967027 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:32:13.966996 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b23fab_1eba_43bf_ae70_bff4daba1fcc.slice/crio-10cb31ebbdfc2e8705c0cea67bc1d42a261981e734861fb66bcf4f56c8672e74 WatchSource:0}: Error finding container 10cb31ebbdfc2e8705c0cea67bc1d42a261981e734861fb66bcf4f56c8672e74: Status 404 returned error can't find the container with id 10cb31ebbdfc2e8705c0cea67bc1d42a261981e734861fb66bcf4f56c8672e74 Oct 30 22:32:13.969997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.969967 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:13.970082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.970032 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:13.970082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:13.970066 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:14.075604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:14.075264 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" event={"ID":"f3b23fab-1eba-43bf-ae70-bff4daba1fcc","Type":"ContainerStarted","Data":"0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30"} Oct 30 22:32:14.075604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:14.075347 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" event={"ID":"f3b23fab-1eba-43bf-ae70-bff4daba1fcc","Type":"ContainerStarted","Data":"10cb31ebbdfc2e8705c0cea67bc1d42a261981e734861fb66bcf4f56c8672e74"} Oct 30 22:32:14.190004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:14.189950 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podStartSLOduration=1.1899342370000001 podStartE2EDuration="1.189934237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:14.189178393 +0000 UTC m=+2754.488588292" watchObservedRunningTime="2026-10-30 22:32:14.189934237 +0000 UTC m=+2754.489344118" Oct 30 22:32:14.475988 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:14.475959 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:15.481974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:15.481940 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:15.928088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:15.928000 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn"] Oct 30 22:32:15.928264 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:15.928212 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" podUID="167b67df-0661-4d48-bdf8-40f82638aa13" containerName="maas-api" containerID="cri-o://bb0bb1346f70b2624e25da725c416b87c51221e1d6856093b227cdfbd3688823" gracePeriod=30 Oct 30 22:32:16.090179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.090142 4991 generic.go:356] "Generic (PLEG): container finished" podID="167b67df-0661-4d48-bdf8-40f82638aa13" containerID="bb0bb1346f70b2624e25da725c416b87c51221e1d6856093b227cdfbd3688823" exitCode=0 Oct 30 22:32:16.090374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.090213 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" event={"ID":"167b67df-0661-4d48-bdf8-40f82638aa13","Type":"ContainerDied","Data":"bb0bb1346f70b2624e25da725c416b87c51221e1d6856093b227cdfbd3688823"} Oct 30 22:32:16.090809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.090794 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:16.091824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.091802 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:32:16.249426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.249398 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:32:16.301115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.301086 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167b67df-0661-4d48-bdf8-40f82638aa13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167b67df-0661-4d48-bdf8-40f82638aa13-openshift-service-ca\") pod \"167b67df-0661-4d48-bdf8-40f82638aa13\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " Oct 30 22:32:16.301300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.301256 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls\") pod \"167b67df-0661-4d48-bdf8-40f82638aa13\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " Oct 30 22:32:16.301421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.301405 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-metrics-certs\") pod \"167b67df-0661-4d48-bdf8-40f82638aa13\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " Oct 30 22:32:16.301490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.301461 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167b67df-0661-4d48-bdf8-40f82638aa13-kube-api-access-252mb\" (UniqueName: \"kubernetes.io/projected/167b67df-0661-4d48-bdf8-40f82638aa13-kube-api-access-252mb\") pod \"167b67df-0661-4d48-bdf8-40f82638aa13\" (UID: \"167b67df-0661-4d48-bdf8-40f82638aa13\") " Oct 30 22:32:16.301566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.301545 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167b67df-0661-4d48-bdf8-40f82638aa13-openshift-service-ca" pod "167b67df-0661-4d48-bdf8-40f82638aa13" (UID: "167b67df-0661-4d48-bdf8-40f82638aa13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:16.302105 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.302082 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167b67df-0661-4d48-bdf8-40f82638aa13-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:16.303985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.303957 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls" pod "167b67df-0661-4d48-bdf8-40f82638aa13" (UID: "167b67df-0661-4d48-bdf8-40f82638aa13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:16.304299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.304272 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167b67df-0661-4d48-bdf8-40f82638aa13-kube-api-access-252mb" pod "167b67df-0661-4d48-bdf8-40f82638aa13" (UID: "167b67df-0661-4d48-bdf8-40f82638aa13"). InnerVolumeSpecName "kube-api-access-252mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:16.304524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.304372 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-metrics-certs" pod "167b67df-0661-4d48-bdf8-40f82638aa13" (UID: "167b67df-0661-4d48-bdf8-40f82638aa13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:16.404103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.404065 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:16.404103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.404106 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167b67df-0661-4d48-bdf8-40f82638aa13-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:16.404346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:16.404123 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252mb\" (UniqueName: \"kubernetes.io/projected/167b67df-0661-4d48-bdf8-40f82638aa13-kube-api-access-252mb\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:17.100606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:17.100577 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" Oct 30 22:32:17.101097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:17.100606 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn" event={"ID":"167b67df-0661-4d48-bdf8-40f82638aa13","Type":"ContainerDied","Data":"1394cbd873bdd77eeff24ad539bf7730e2e4c97224b2cd40336128baa5cc54d3"} Oct 30 22:32:17.101097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:17.100683 4991 scope.go:122] "RemoveContainer" containerID="bb0bb1346f70b2624e25da725c416b87c51221e1d6856093b227cdfbd3688823" Oct 30 22:32:17.262199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:17.262169 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn"] Oct 30 22:32:17.309561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:17.309526 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c301cf-58f5dfb855-f4pkn"] Oct 30 22:32:17.877786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:17.877748 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167b67df-0661-4d48-bdf8-40f82638aa13" path="/var/lib/kubelet/pods/167b67df-0661-4d48-bdf8-40f82638aa13/volumes" Oct 30 22:32:17.912511 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:32:17.912474 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:32:27.884606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.884566 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j"] Oct 30 22:32:27.885399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.885376 4991 cpu_manager.go:414] "containerMap: removing container" podUID="167b67df-0661-4d48-bdf8-40f82638aa13" containerName="maas-api" Oct 30 22:32:27.885399 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.885399 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167b67df-0661-4d48-bdf8-40f82638aa13" containerName="maas-api" Oct 30 22:32:27.885593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.885579 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="167b67df-0661-4d48-bdf8-40f82638aa13" containerName="maas-api" Oct 30 22:32:27.889363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.889337 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:27.986270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.986233 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j"] Oct 30 22:32:27.991486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.990552 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-credential-socket\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:27.991636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.991531 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-socket\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:27.991636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.991571 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-podinfo\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:27.991636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.991602 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzvp\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-kube-api-access-6kzvp\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.005861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.991681 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istiod-ca-cert\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.005861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.991733 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-token\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.005861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.991891 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-certs\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.005861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.991925 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-data\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.005861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:27.992006 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-envoy\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.015287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.015259 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-3b392d-openshift-default-dockercfg-hwsrh\"" Oct 30 22:32:28.095674 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.094180 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-certs\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097459 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-data\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097564 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-envoy\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097731 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-credential-socket\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097767 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-socket\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097814 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-podinfo\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097850 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzvp\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-kube-api-access-6kzvp\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097890 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istiod-ca-cert\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.097956 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-token\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.094899 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-certs\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.098686 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-data\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.098703 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-socket\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.099210 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-credential-socket\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.099592 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istiod-ca-cert\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.102943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.102664 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-envoy\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.108705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.105383 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-podinfo\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.156358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.156273 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzvp\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-kube-api-access-6kzvp\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.156520 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.156490 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-token\") pod \"e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.203275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.203243 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:28.677761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.677724 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j"] Oct 30 22:32:28.683596 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:32:28.683561 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843f57c2_8883_4f2a_aec5_3e8e940ad5fd.slice/crio-b1c1758abf621a6984c6796bb0366469b8db25a31cb3dae0c296171cffca9474 WatchSource:0}: Error finding container b1c1758abf621a6984c6796bb0366469b8db25a31cb3dae0c296171cffca9474: Status 404 returned error can't find the container with id b1c1758abf621a6984c6796bb0366469b8db25a31cb3dae0c296171cffca9474 Oct 30 22:32:28.686298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.686265 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:28.686434 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.686362 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:28.686434 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:28.686397 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:32:29.194957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:29.194921 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" event={"ID":"843f57c2-8883-4f2a-aec5-3e8e940ad5fd","Type":"ContainerStarted","Data":"ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba"} Oct 30 22:32:29.195412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:29.194977 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" event={"ID":"843f57c2-8883-4f2a-aec5-3e8e940ad5fd","Type":"ContainerStarted","Data":"b1c1758abf621a6984c6796bb0366469b8db25a31cb3dae0c296171cffca9474"} Oct 30 22:32:29.204281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:29.204259 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:29.209752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:29.209731 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:29.312576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:29.312532 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podStartSLOduration=2.31251991 podStartE2EDuration="2.31251991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:29.309493697 +0000 UTC m=+2769.608903604" watchObservedRunningTime="2026-10-30 22:32:29.31251991 +0000 UTC m=+2769.611929790" Oct 30 22:32:30.202635 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:30.202598 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:30.203430 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:32:30.203411 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:32:47.911983 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:32:47.911945 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:33:17.914240 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:33:17.914205 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:33:25.633062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:25.632974 4991 generic.go:356] "Generic (PLEG): container finished" podID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerID="7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83" exitCode=0 Oct 30 22:33:25.633062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:25.633047 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" event={"ID":"39de1aba-6701-42bf-bcfd-d2327f97f5a5","Type":"ContainerDied","Data":"7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83"} Oct 30 22:33:25.633545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:25.633515 4991 scope.go:122] "RemoveContainer" containerID="7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83" Oct 30 22:33:25.635251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:25.635219 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:25.635421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:25.635290 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:25.635421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:25.635362 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:33:26.660750 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:26.660703 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" event={"ID":"39de1aba-6701-42bf-bcfd-d2327f97f5a5","Type":"ContainerStarted","Data":"53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a"} Oct 30 22:33:27.268557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:27.268525 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:33:27.268754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:27.268706 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:33:27.268800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:27.268751 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:33:28.268190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:28.268158 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:33:28.268631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:28.268217 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:33:29.268541 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:29.268501 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:33:29.269097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:29.269070 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:33:30.268681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:30.268579 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:33:30.268681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:30.268639 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:33:31.268192 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:31.268159 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:33:31.268395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:31.268205 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 22:33:32.272448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:32.272419 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:33:32.272932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:32.272914 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:33:32.273565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:32.273548 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:33:47.965043 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:33:47.965001 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:33:48.278107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.278075 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9"] Oct 30 22:33:48.284595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.284563 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.339104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.339072 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9"] Oct 30 22:33:48.380846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.380817 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-435923-serving-cert\"" Oct 30 22:33:48.430944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.430895 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92343dce-b0c6-44f8-a913-e718e6eb0ad7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.431109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.430946 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.431185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.431129 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.431423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.431397 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcpt\" (UniqueName: \"kubernetes.io/projected/92343dce-b0c6-44f8-a913-e718e6eb0ad7-kube-api-access-kvcpt\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.534523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.534415 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcpt\" (UniqueName: \"kubernetes.io/projected/92343dce-b0c6-44f8-a913-e718e6eb0ad7-kube-api-access-kvcpt\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.534882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.534850 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92343dce-b0c6-44f8-a913-e718e6eb0ad7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.535025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.534897 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.535092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.535039 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.535590 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:33:48.535553 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-serving-cert: secret "maas-api-e2e-shared-b-w3-435923-serving-cert" not found Oct 30 22:33:48.535716 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:33:48.535680 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls podName:92343dce-b0c6-44f8-a913-e718e6eb0ad7 nodeName:}" failed. No retries permitted until 2026-10-30 22:33:49.035655037 +0000 UTC m=+2849.335064918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" (UID: "92343dce-b0c6-44f8-a913-e718e6eb0ad7") : secret "maas-api-e2e-shared-b-w3-435923-serving-cert" not found Oct 30 22:33:48.535884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.535859 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92343dce-b0c6-44f8-a913-e718e6eb0ad7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.537871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.537849 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:48.595947 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:48.595918 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcpt\" (UniqueName: \"kubernetes.io/projected/92343dce-b0c6-44f8-a913-e718e6eb0ad7-kube-api-access-kvcpt\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:49.045743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:49.045705 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:49.048117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:49.048096 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:49.201303 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:49.201276 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:49.585949 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:49.585913 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9"] Oct 30 22:33:49.591930 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:33:49.591902 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92343dce_b0c6_44f8_a913_e718e6eb0ad7.slice/crio-52c66885bf3555f03b15865e295c3e66ecd7185d6e4af26ab821dd90340d252e WatchSource:0}: Error finding container 52c66885bf3555f03b15865e295c3e66ecd7185d6e4af26ab821dd90340d252e: Status 404 returned error can't find the container with id 52c66885bf3555f03b15865e295c3e66ecd7185d6e4af26ab821dd90340d252e Oct 30 22:33:49.593332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:49.593298 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:33:49.973433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:49.972246 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" event={"ID":"92343dce-b0c6-44f8-a913-e718e6eb0ad7","Type":"ContainerStarted","Data":"52c66885bf3555f03b15865e295c3e66ecd7185d6e4af26ab821dd90340d252e"} Oct 30 22:33:50.990525 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:50.990490 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" event={"ID":"92343dce-b0c6-44f8-a913-e718e6eb0ad7","Type":"ContainerStarted","Data":"698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3"} Oct 30 22:33:50.990999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:50.990982 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:33:51.125345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:51.120033 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" podStartSLOduration=2.210940889 podStartE2EDuration="3.120010212s" totalImagesPullingTime="909.069323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:51.119121437 +0000 UTC m=+2851.418531342" watchObservedRunningTime="2026-10-30 22:33:51.120010212 +0000 UTC m=+2851.419420094" Oct 30 22:33:58.011089 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:33:58.011050 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:34:00.064528 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.064429 4991 generic.go:356] "Generic (PLEG): container finished" podID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerID="ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba" exitCode=0 Oct 30 22:34:00.064528 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.064504 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" event={"ID":"843f57c2-8883-4f2a-aec5-3e8e940ad5fd","Type":"ContainerDied","Data":"ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba"} Oct 30 22:34:00.065025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.064897 4991 scope.go:122] "RemoveContainer" containerID="ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba" Oct 30 22:34:00.066652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.066616 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:34:00.066802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.066700 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:34:00.066802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.066744 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:34:00.067193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.067163 4991 generic.go:356] "Generic (PLEG): container finished" podID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerID="0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30" exitCode=0 Oct 30 22:34:00.067370 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.067205 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" event={"ID":"f3b23fab-1eba-43bf-ae70-bff4daba1fcc","Type":"ContainerDied","Data":"0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30"} Oct 30 22:34:00.067560 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.067543 4991 scope.go:122] "RemoveContainer" containerID="0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30" Oct 30 22:34:00.069088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.069054 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:34:00.069190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.069140 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:34:00.069252 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.069189 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:34:00.203760 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:00.203428 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:34:01.086569 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.085066 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" event={"ID":"843f57c2-8883-4f2a-aec5-3e8e940ad5fd","Type":"ContainerStarted","Data":"750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4"} Oct 30 22:34:01.090142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.088915 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" event={"ID":"f3b23fab-1eba-43bf-ae70-bff4daba1fcc","Type":"ContainerStarted","Data":"5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75"} Oct 30 22:34:01.091422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.091399 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:34:01.204454 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.204382 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:34:01.205504 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.205470 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:34:01.205659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.205525 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:34:01.484339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.480797 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:34:01.484339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.480944 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:34:01.484339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:01.480999 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:34:02.203978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:02.203945 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:34:02.204401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:02.203995 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:34:02.475373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:02.475337 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:34:02.475659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:02.475631 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:34:03.204485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:03.204450 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:34:03.204877 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:03.204506 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:34:03.476026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:03.475992 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:34:03.476193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:03.476045 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:34:04.203679 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:04.203638 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:34:04.203864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:04.203690 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:34:04.475871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:04.475838 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:34:04.476508 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:04.476461 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:34:05.204281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:05.204246 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:34:05.204554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:05.204311 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:34:05.476221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:05.476191 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:34:05.476656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:05.476248 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:34:06.204601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:06.204563 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:34:06.204800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:06.204619 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:34:06.475600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:06.475566 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:34:06.475800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:06.475622 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:34:07.209191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:07.209162 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:34:07.209711 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:07.209654 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:34:07.210396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:07.210377 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:34:07.479888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:07.479858 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:34:07.480765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:07.480745 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:34:17.910602 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:34:17.910563 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:34:26.982613 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:26.982581 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz"] Oct 30 22:34:26.987062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:26.987044 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.084688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.084658 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9717a8\"/\"default-dockercfg-6f4hc\"" Oct 30 22:34:27.084688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.084658 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9717a8\"/\"openshift-service-ca.crt\"" Oct 30 22:34:27.084953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.084657 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9717a8\"/\"kube-root-ca.crt\"" Oct 30 22:34:27.085392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.085371 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9717a8\"/\"auth-test-model-9717a8-kserve-self-signed-certs\"" Oct 30 22:34:27.088612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.088588 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz"] Oct 30 22:34:27.100464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.100437 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-tmp-dir\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.100553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.100480 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-dshm\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.100598 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.100576 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96db307-5896-4311-93e0-4946a617469f-tls-certs\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.100777 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.100759 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-home\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.100814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.100790 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-model-cache\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.100902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.100888 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49f2\" (UniqueName: \"kubernetes.io/projected/c96db307-5896-4311-93e0-4946a617469f-kube-api-access-t49f2\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202180 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96db307-5896-4311-93e0-4946a617469f-tls-certs\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202479 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202429 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-home\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202491 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-model-cache\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202558 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49f2\" (UniqueName: \"kubernetes.io/projected/c96db307-5896-4311-93e0-4946a617469f-kube-api-access-t49f2\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202772 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-tmp-dir\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202873 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202842 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-dshm\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202909 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-home\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.202996 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.202972 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-model-cache\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.203265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.203242 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-tmp-dir\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.204961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.204938 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96db307-5896-4311-93e0-4946a617469f-tls-certs\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.205090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.205056 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-dshm\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.266369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.266280 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49f2\" (UniqueName: \"kubernetes.io/projected/c96db307-5896-4311-93e0-4946a617469f-kube-api-access-t49f2\") pod \"auth-test-model-9717a8-kserve-678787c4f7-crjcz\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.299221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.299196 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:27.683737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:27.683661 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz"] Oct 30 22:34:27.689090 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:34:27.689063 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc96db307_5896_4311_93e0_4946a617469f.slice/crio-0922922ab561a84799673a5d74167e06ab77fbe7dd726b95849a9aaedc221812 WatchSource:0}: Error finding container 0922922ab561a84799673a5d74167e06ab77fbe7dd726b95849a9aaedc221812: Status 404 returned error can't find the container with id 0922922ab561a84799673a5d74167e06ab77fbe7dd726b95849a9aaedc221812 Oct 30 22:34:28.305962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:28.305925 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" event={"ID":"c96db307-5896-4311-93e0-4946a617469f","Type":"ContainerStarted","Data":"33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903"} Oct 30 22:34:28.305962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:28.305975 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" event={"ID":"c96db307-5896-4311-93e0-4946a617469f","Type":"ContainerStarted","Data":"0922922ab561a84799673a5d74167e06ab77fbe7dd726b95849a9aaedc221812"} Oct 30 22:34:28.434422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:28.434365 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" podStartSLOduration=2.434346846 podStartE2EDuration="2.434346846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:28.429140168 +0000 UTC m=+2888.728550067" watchObservedRunningTime="2026-10-30 22:34:28.434346846 +0000 UTC m=+2888.733756728" Oct 30 22:34:29.300252 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:29.300219 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:37.299587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:37.299542 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:37.312472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:37.312439 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:37.412484 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:34:37.412449 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:34:47.911105 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:34:47.911066 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:35:17.912709 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:35:17.912675 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:35:35.832724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.832686 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn"] Oct 30 22:35:35.838697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.838675 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.883965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.883929 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6lg\" (UniqueName: \"kubernetes.io/projected/38eb7556-0637-41b8-9d8e-c6d6773f76ca-kube-api-access-nb6lg\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.884120 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.883978 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-home\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.884194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.884172 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-model-cache\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.884249 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.884222 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tls-certs\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.884415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.884391 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-dshm\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.884491 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.884460 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tmp-dir\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.891694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.891669 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn"] Oct 30 22:35:35.933098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.933074 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f39eb7\"/\"kube-root-ca.crt\"" Oct 30 22:35:35.933306 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.933098 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f39eb7\"/\"openshift-service-ca.crt\"" Oct 30 22:35:35.933306 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.933194 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f39eb7\"/\"default-dockercfg-9lpk2\"" Oct 30 22:35:35.934626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.934601 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f39eb7\"/\"auth-test-model-f39eb7-kserve-self-signed-certs\"" Oct 30 22:35:35.986449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.986422 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-model-cache\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.986607 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.986478 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tls-certs\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.986607 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.986531 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-dshm\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.986607 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.986584 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tmp-dir\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.986782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.986752 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6lg\" (UniqueName: \"kubernetes.io/projected/38eb7556-0637-41b8-9d8e-c6d6773f76ca-kube-api-access-nb6lg\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.986847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.986783 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-home\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.986904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.986888 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-model-cache\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.987075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.987050 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tmp-dir\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.987172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.987145 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-home\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.988912 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.988891 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-dshm\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:35.989793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:35.989773 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tls-certs\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:36.049742 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:36.049708 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6lg\" (UniqueName: \"kubernetes.io/projected/38eb7556-0637-41b8-9d8e-c6d6773f76ca-kube-api-access-nb6lg\") pod \"auth-test-model-f39eb7-kserve-7d5b5589-8fvpn\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:36.153818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:36.153731 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:36.552679 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:36.552645 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn"] Oct 30 22:35:36.560466 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:35:36.560425 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38eb7556_0637_41b8_9d8e_c6d6773f76ca.slice/crio-782c64de7ca4d87565049ac69c17b92b663e7318f1dc4dcfc012da473f692d77 WatchSource:0}: Error finding container 782c64de7ca4d87565049ac69c17b92b663e7318f1dc4dcfc012da473f692d77: Status 404 returned error can't find the container with id 782c64de7ca4d87565049ac69c17b92b663e7318f1dc4dcfc012da473f692d77 Oct 30 22:35:36.889428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:36.888626 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" event={"ID":"38eb7556-0637-41b8-9d8e-c6d6773f76ca","Type":"ContainerStarted","Data":"16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286"} Oct 30 22:35:36.889428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:36.888675 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" event={"ID":"38eb7556-0637-41b8-9d8e-c6d6773f76ca","Type":"ContainerStarted","Data":"782c64de7ca4d87565049ac69c17b92b663e7318f1dc4dcfc012da473f692d77"} Oct 30 22:35:37.007186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:37.007131 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" podStartSLOduration=2.007115705 podStartE2EDuration="2.007115705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:37.005997574 +0000 UTC m=+2957.305407453" watchObservedRunningTime="2026-10-30 22:35:37.007115705 +0000 UTC m=+2957.306525585" Oct 30 22:35:37.154761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:37.154678 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:43.797850 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:43.797812 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft"] Oct 30 22:35:46.154517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:46.154485 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:46.166808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:46.166781 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:46.978371 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:46.978345 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:35:47.912595 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:35:47.912562 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:35:58.819300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:58.819261 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" podUID="8129b506-3ca3-489f-8b34-c9bfd438b93e" containerName="main" containerID="cri-o://cf0ba964f8d3156d65db39da9043e063f66f1fe2e5071f27a92b94caf7c09135" gracePeriod=45 Oct 30 22:35:59.053251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.053216 4991 generic.go:356] "Generic (PLEG): container finished" podID="8129b506-3ca3-489f-8b34-c9bfd438b93e" containerID="cf0ba964f8d3156d65db39da9043e063f66f1fe2e5071f27a92b94caf7c09135" exitCode=0 Oct 30 22:35:59.053491 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.053469 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" event={"ID":"8129b506-3ca3-489f-8b34-c9bfd438b93e","Type":"ContainerDied","Data":"cf0ba964f8d3156d65db39da9043e063f66f1fe2e5071f27a92b94caf7c09135"} Oct 30 22:35:59.112976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.112949 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:35:59.244698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.244665 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-tmp-dir\") pod \"8129b506-3ca3-489f-8b34-c9bfd438b93e\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " Oct 30 22:35:59.244888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.244732 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8129b506-3ca3-489f-8b34-c9bfd438b93e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8129b506-3ca3-489f-8b34-c9bfd438b93e-tls-certs\") pod \"8129b506-3ca3-489f-8b34-c9bfd438b93e\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " Oct 30 22:35:59.245040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.245017 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-tmp-dir" pod "8129b506-3ca3-489f-8b34-c9bfd438b93e" (UID: "8129b506-3ca3-489f-8b34-c9bfd438b93e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:59.245173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.245141 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8129b506-3ca3-489f-8b34-c9bfd438b93e-kube-api-access-5cjmp\" (UniqueName: \"kubernetes.io/projected/8129b506-3ca3-489f-8b34-c9bfd438b93e-kube-api-access-5cjmp\") pod \"8129b506-3ca3-489f-8b34-c9bfd438b93e\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " Oct 30 22:35:59.245459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.245440 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-dshm\") pod \"8129b506-3ca3-489f-8b34-c9bfd438b93e\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " Oct 30 22:35:59.245812 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.245792 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-model-cache\") pod \"8129b506-3ca3-489f-8b34-c9bfd438b93e\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " Oct 30 22:35:59.246238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.246218 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-home\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-home\") pod \"8129b506-3ca3-489f-8b34-c9bfd438b93e\" (UID: \"8129b506-3ca3-489f-8b34-c9bfd438b93e\") " Oct 30 22:35:59.246578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.246445 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-model-cache" pod "8129b506-3ca3-489f-8b34-c9bfd438b93e" (UID: "8129b506-3ca3-489f-8b34-c9bfd438b93e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:59.246578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.246562 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-home" pod "8129b506-3ca3-489f-8b34-c9bfd438b93e" (UID: "8129b506-3ca3-489f-8b34-c9bfd438b93e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:59.247420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.247399 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.247552 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.247536 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.247652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.247640 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.249582 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.249559 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8129b506-3ca3-489f-8b34-c9bfd438b93e-tls-certs" pod "8129b506-3ca3-489f-8b34-c9bfd438b93e" (UID: "8129b506-3ca3-489f-8b34-c9bfd438b93e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:59.249688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.249645 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-dshm" pod "8129b506-3ca3-489f-8b34-c9bfd438b93e" (UID: "8129b506-3ca3-489f-8b34-c9bfd438b93e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:59.266909 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.266737 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8129b506-3ca3-489f-8b34-c9bfd438b93e-kube-api-access-5cjmp" pod "8129b506-3ca3-489f-8b34-c9bfd438b93e" (UID: "8129b506-3ca3-489f-8b34-c9bfd438b93e"). InnerVolumeSpecName "kube-api-access-5cjmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:59.349600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.349501 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129b506-3ca3-489f-8b34-c9bfd438b93e-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.349600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.349537 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cjmp\" (UniqueName: \"kubernetes.io/projected/8129b506-3ca3-489f-8b34-c9bfd438b93e-kube-api-access-5cjmp\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.349600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:35:59.349563 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129b506-3ca3-489f-8b34-c9bfd438b93e-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:00.095266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:00.094406 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" event={"ID":"8129b506-3ca3-489f-8b34-c9bfd438b93e","Type":"ContainerDied","Data":"0cc8c12e212b78c5bf9fe050c3ea2de5ac502a87811e7870fad71e0ef14665bd"} Oct 30 22:36:00.095266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:00.094480 4991 scope.go:122] "RemoveContainer" containerID="cf0ba964f8d3156d65db39da9043e063f66f1fe2e5071f27a92b94caf7c09135" Oct 30 22:36:00.095266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:00.094643 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft" Oct 30 22:36:00.258674 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:00.258639 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft"] Oct 30 22:36:00.304291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:00.304257 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-1875b4/praxis-route-1875b4-kserve-5fc4b79f4c-fjpft"] Oct 30 22:36:01.910215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:01.910115 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8129b506-3ca3-489f-8b34-c9bfd438b93e" path="/var/lib/kubelet/pods/8129b506-3ca3-489f-8b34-c9bfd438b93e/volumes" Oct 30 22:36:05.480672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:05.480642 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8"] Oct 30 22:36:05.481211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:05.480856 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" containerID="cri-o://172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46" gracePeriod=30 Oct 30 22:36:08.364333 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:08.364295 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv"] Oct 30 22:36:08.364815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:08.364572 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" containerID="cri-o://c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1" gracePeriod=30 Oct 30 22:36:09.984222 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:09.984188 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:09.984643 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:09.984242 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:10.789151 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.789124 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:36:10.833245 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833208 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-data\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833303 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-podinfo\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833340 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-socket\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833368 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-credential-socket\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833413 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-certs\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833469 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-envoy\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833521 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-data" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:10.833683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833581 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-token\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-token\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833641 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c585d34-7460-440f-bbb4-c00a776e52ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c585d34-7460-440f-bbb4-c00a776e52ba-istiod-ca-cert\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833659 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-socket" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:10.833926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833769 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-kube-api-access-767c2\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-kube-api-access-767c2\") pod \"0c585d34-7460-440f-bbb4-c00a776e52ba\" (UID: \"0c585d34-7460-440f-bbb4-c00a776e52ba\") " Oct 30 22:36:10.833926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833779 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-certs" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:10.834034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.833997 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-credential-socket" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:10.834091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.834075 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c585d34-7460-440f-bbb4-c00a776e52ba-istiod-ca-cert" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:10.834469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.834435 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c585d34-7460-440f-bbb4-c00a776e52ba-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.834593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.834471 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.834593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.834486 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.834593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.834527 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.834593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.834545 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.835787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.835762 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-podinfo" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:10.836098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.836080 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-kube-api-access-767c2" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "kube-api-access-767c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:10.836401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.836384 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-token" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:10.836651 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.836623 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-envoy" pod "0c585d34-7460-440f-bbb4-c00a776e52ba" (UID: "0c585d34-7460-440f-bbb4-c00a776e52ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:10.935409 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.935373 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-767c2\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-kube-api-access-767c2\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.935409 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.935407 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.935648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.935422 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:10.935648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:10.935438 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c585d34-7460-440f-bbb4-c00a776e52ba-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.196495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.196407 4991 generic.go:356] "Generic (PLEG): container finished" podID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerID="172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46" exitCode=0 Oct 30 22:36:11.196495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.196457 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" Oct 30 22:36:11.196975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.196496 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" event={"ID":"0c585d34-7460-440f-bbb4-c00a776e52ba","Type":"ContainerDied","Data":"172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46"} Oct 30 22:36:11.196975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.196558 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8" event={"ID":"0c585d34-7460-440f-bbb4-c00a776e52ba","Type":"ContainerDied","Data":"9ac55a05a2cafc501543623fc3dd252444862b7542e5a8cb5e3e3c8e6d08cdcb"} Oct 30 22:36:11.196975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.196589 4991 scope.go:122] "RemoveContainer" containerID="172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46" Oct 30 22:36:11.208313 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.208298 4991 scope.go:122] "RemoveContainer" containerID="172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46" Oct 30 22:36:11.208612 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:11.208595 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46\": container with ID starting with 172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46 not found: ID does not exist" containerID="172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46" Oct 30 22:36:11.208661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.208620 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46"} err="failed to get container status \"172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46\": rpc error: code = NotFound desc = could not find container \"172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46\": container with ID starting with 172aa206db9335dff5a5c53ed708bd89850e1f08ddda5bfe6008786c6a485e46 not found: ID does not exist" Oct 30 22:36:11.353313 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.353281 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8"] Oct 30 22:36:11.357507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.357481 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4bgb8"] Oct 30 22:36:11.876801 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:11.876770 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" path="/var/lib/kubelet/pods/0c585d34-7460-440f-bbb4-c00a776e52ba/volumes" Oct 30 22:36:13.374496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.374454 4991 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": EOF" start-of-body= Oct 30 22:36:13.375297 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.375160 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": EOF" Oct 30 22:36:13.643899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.643839 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:36:13.771479 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771451 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-credential-socket\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771490 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-kube-api-access-9sgcx\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-kube-api-access-9sgcx\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771564 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-socket\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771617 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-podinfo\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771674 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-data\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771700 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-credential-socket" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:13.771768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771724 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-certs\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771752 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-envoy\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771793 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istiod-ca-cert\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771943 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-token\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-token\") pod \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\" (UID: \"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c\") " Oct 30 22:36:13.771982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771949 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-socket" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:13.771982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771966 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-data" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:13.772208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.771981 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-certs" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:13.772269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.772240 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istiod-ca-cert" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:13.772517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.772484 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.772517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.772508 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.772685 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.772521 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.772685 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.772535 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.772685 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.772549 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.774066 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.774039 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-token" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:13.774161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.774125 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-podinfo" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:13.774367 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.774345 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-envoy" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:13.774429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.774347 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-kube-api-access-9sgcx" pod "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" (UID: "ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c"). InnerVolumeSpecName "kube-api-access-9sgcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:13.873564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.873533 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.873564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.873560 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.873564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.873570 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sgcx\" (UniqueName: \"kubernetes.io/projected/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-kube-api-access-9sgcx\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:13.873817 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:13.873582 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:14.221907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.221870 4991 generic.go:356] "Generic (PLEG): container finished" podID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerID="c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1" exitCode=0 Oct 30 22:36:14.222087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.221949 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" Oct 30 22:36:14.222087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.221963 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" event={"ID":"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c","Type":"ContainerDied","Data":"c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1"} Oct 30 22:36:14.222087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.222060 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv" event={"ID":"ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c","Type":"ContainerDied","Data":"efafc3d564447c39f3f24f8b0660bfb3c5b108fcab3278f23279abb3306d4606"} Oct 30 22:36:14.222226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.222091 4991 scope.go:122] "RemoveContainer" containerID="c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1" Oct 30 22:36:14.233215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.233175 4991 scope.go:122] "RemoveContainer" containerID="7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27" Oct 30 22:36:14.242379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.242363 4991 scope.go:122] "RemoveContainer" containerID="c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1" Oct 30 22:36:14.242640 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:14.242624 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1\": container with ID starting with c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1 not found: ID does not exist" containerID="c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1" Oct 30 22:36:14.242693 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.242648 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1"} err="failed to get container status \"c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1\": rpc error: code = NotFound desc = could not find container \"c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1\": container with ID starting with c9f7f2ad1cecead42becfbd9980eef55dfce15269fdadf72328b5cda24eae7c1 not found: ID does not exist" Oct 30 22:36:14.242693 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.242667 4991 scope.go:122] "RemoveContainer" containerID="7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27" Oct 30 22:36:14.242889 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:14.242871 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27\": container with ID starting with 7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27 not found: ID does not exist" containerID="7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27" Oct 30 22:36:14.242944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.242898 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27"} err="failed to get container status \"7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27\": rpc error: code = NotFound desc = could not find container \"7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27\": container with ID starting with 7bf05be30b3b9ebe5e5c7b71aae0bf016705b8a4b1230e163ff8f3f9d9953d27 not found: ID does not exist" Oct 30 22:36:14.379551 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.379513 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv"] Oct 30 22:36:14.425665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:14.425633 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-nxncv"] Oct 30 22:36:15.875298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:15.875256 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" path="/var/lib/kubelet/pods/ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c/volumes" Oct 30 22:36:16.689712 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:16.689679 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh"] Oct 30 22:36:16.689918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:16.689875 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" containerID="cri-o://d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26" gracePeriod=30 Oct 30 22:36:16.736339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:16.736294 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt"] Oct 30 22:36:16.736519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:16.736496 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" containerID="cri-o://19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1" gracePeriod=30 Oct 30 22:36:17.911508 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:17.911475 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:36:19.141405 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.141367 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm"] Oct 30 22:36:19.141976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.141960 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" Oct 30 22:36:19.141976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.141975 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" Oct 30 22:36:19.142098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.141991 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8129b506-3ca3-489f-8b34-c9bfd438b93e" containerName="main" Oct 30 22:36:19.142098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.141997 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8129b506-3ca3-489f-8b34-c9bfd438b93e" containerName="main" Oct 30 22:36:19.142098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.142017 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" Oct 30 22:36:19.142098 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.142022 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" Oct 30 22:36:19.142315 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.142132 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" Oct 30 22:36:19.142315 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.142141 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8129b506-3ca3-489f-8b34-c9bfd438b93e" containerName="main" Oct 30 22:36:19.142315 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.142151 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c585d34-7460-440f-bbb4-c00a776e52ba" containerName="istio-proxy" Oct 30 22:36:19.145614 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.145592 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.224524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.224491 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm"] Oct 30 22:36:19.238663 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.238635 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-model-cache\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.238793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.238674 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-dshm\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.238793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.238710 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-tmp-dir\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.238880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.238849 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmz64\" (UniqueName: \"kubernetes.io/projected/54e9b919-9860-4ace-9411-fc163a74b3e9-kube-api-access-lmz64\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.238968 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.238955 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e9b919-9860-4ace-9411-fc163a74b3e9-tls-certs\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.239007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.238987 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-home\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.249164 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.249141 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-68f80c\"/\"openshift-service-ca.crt\"" Oct 30 22:36:19.249164 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.249140 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-68f80c\"/\"kube-root-ca.crt\"" Oct 30 22:36:19.249451 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.249438 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-68f80c\"/\"default-dockercfg-h9hdw\"" Oct 30 22:36:19.249542 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.249526 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-68f80c\"/\"auth-test-model-68f80c-kserve-self-signed-certs\"" Oct 30 22:36:19.340931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.340874 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmz64\" (UniqueName: \"kubernetes.io/projected/54e9b919-9860-4ace-9411-fc163a74b3e9-kube-api-access-lmz64\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.341138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341025 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e9b919-9860-4ace-9411-fc163a74b3e9-tls-certs\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.341138 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341072 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-home\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.341275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341174 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-model-cache\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.341275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341214 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-dshm\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.341275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341269 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-tmp-dir\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.344110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341728 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-home\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.344110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341742 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-tmp-dir\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.344110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.341986 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-model-cache\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.348149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.348125 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-dshm\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.351280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.351258 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e9b919-9860-4ace-9411-fc163a74b3e9-tls-certs\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.399622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.399555 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmz64\" (UniqueName: \"kubernetes.io/projected/54e9b919-9860-4ace-9411-fc163a74b3e9-kube-api-access-lmz64\") pod \"auth-test-model-68f80c-kserve-bbf78c949-49wbm\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.461418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.461390 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:19.840420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:19.840386 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm"] Oct 30 22:36:19.849733 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:36:19.849705 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54e9b919_9860_4ace_9411_fc163a74b3e9.slice/crio-6b000116a66ec8fe2bc7f40e8a9d8cbe7cde78c231a240581a816d4ec072b229 WatchSource:0}: Error finding container 6b000116a66ec8fe2bc7f40e8a9d8cbe7cde78c231a240581a816d4ec072b229: Status 404 returned error can't find the container with id 6b000116a66ec8fe2bc7f40e8a9d8cbe7cde78c231a240581a816d4ec072b229 Oct 30 22:36:20.276187 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.276142 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" event={"ID":"54e9b919-9860-4ace-9411-fc163a74b3e9","Type":"ContainerStarted","Data":"3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72"} Oct 30 22:36:20.276620 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.276195 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" event={"ID":"54e9b919-9860-4ace-9411-fc163a74b3e9","Type":"ContainerStarted","Data":"6b000116a66ec8fe2bc7f40e8a9d8cbe7cde78c231a240581a816d4ec072b229"} Oct 30 22:36:20.393255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.393196 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" podStartSLOduration=1.393183284 podStartE2EDuration="1.393183284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:20.38917241 +0000 UTC m=+3000.688582312" watchObservedRunningTime="2026-10-30 22:36:20.393183284 +0000 UTC m=+3000.692593163" Oct 30 22:36:20.422634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.422601 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:36:20.435935 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.435910 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:36:20.462125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.462100 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:20.631580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.624340 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv"] Oct 30 22:36:20.631580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.624615 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" containerID="cri-o://3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c" gracePeriod=30 Oct 30 22:36:20.631580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.628637 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb"] Oct 30 22:36:20.631580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:20.628833 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" containerID="cri-o://54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec" gracePeriod=30 Oct 30 22:36:21.707043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:21.707006 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": EOF" start-of-body= Oct 30 22:36:21.707545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:21.707073 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": EOF" Oct 30 22:36:22.020144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.020117 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:36:22.045049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.045028 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:36:22.188706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.188673 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-socket\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.188866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.188726 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-data\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.188866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.188762 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02424f4-1f21-461e-9f87-dd21407f5292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02424f4-1f21-461e-9f87-dd21407f5292-istiod-ca-cert\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.188866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.188808 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-podinfo\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.189049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.188945 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-istio-token\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-istio-token\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.189049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189008 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-credential-socket\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.189049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.188953 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-socket" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.189218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189057 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-credential-socket\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.189218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189083 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09394e12-bb87-4a21-891a-5fdae7e11eb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09394e12-bb87-4a21-891a-5fdae7e11eb1-istiod-ca-cert\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.189218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189117 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-token\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-token\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.189218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189166 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-data\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.189218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189211 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-socket\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189248 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-kube-api-access-84jv4\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-kube-api-access-84jv4\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189168 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02424f4-1f21-461e-9f87-dd21407f5292-istiod-ca-cert" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189265 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-credential-socket" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189369 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c02424f4-1f21-461e-9f87-dd21407f5292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02424f4-1f21-461e-9f87-dd21407f5292-istio-podinfo\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189429 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-envoy\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189458 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-envoy\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189456 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-credential-socket" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189466 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-data" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189489 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-socket" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.189539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189497 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-certs\") pod \"09394e12-bb87-4a21-891a-5fdae7e11eb1\" (UID: \"09394e12-bb87-4a21-891a-5fdae7e11eb1\") " Oct 30 22:36:22.190119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189542 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-certs\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.190119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189683 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-kube-api-access-84g7j\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-kube-api-access-84g7j\") pod \"c02424f4-1f21-461e-9f87-dd21407f5292\" (UID: \"c02424f4-1f21-461e-9f87-dd21407f5292\") " Oct 30 22:36:22.190119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189718 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09394e12-bb87-4a21-891a-5fdae7e11eb1-istiod-ca-cert" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:22.190119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.189951 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-certs" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.190119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190031 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-certs" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.190119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190072 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-data" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.190457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190392 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190415 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190432 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190446 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190471 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190487 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190501 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02424f4-1f21-461e-9f87-dd21407f5292-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190515 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190530 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.190670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.190544 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09394e12-bb87-4a21-891a-5fdae7e11eb1-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.191694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.191636 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-istio-token" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:22.192250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.192225 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-envoy" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.192395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.192248 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c02424f4-1f21-461e-9f87-dd21407f5292-istio-podinfo" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:22.192395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.192368 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-token" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:22.192488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.192441 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-kube-api-access-84jv4" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "kube-api-access-84jv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:22.192488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.192465 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-kube-api-access-84g7j" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "kube-api-access-84g7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:22.193040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.192612 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-podinfo" pod "09394e12-bb87-4a21-891a-5fdae7e11eb1" (UID: "09394e12-bb87-4a21-891a-5fdae7e11eb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:22.193460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.193445 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-envoy" pod "c02424f4-1f21-461e-9f87-dd21407f5292" (UID: "c02424f4-1f21-461e-9f87-dd21407f5292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:22.291505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291477 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.291505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291501 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.291505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291510 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.291839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291520 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84jv4\" (UniqueName: \"kubernetes.io/projected/09394e12-bb87-4a21-891a-5fdae7e11eb1-kube-api-access-84jv4\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.291839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291528 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02424f4-1f21-461e-9f87-dd21407f5292-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.291839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291537 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09394e12-bb87-4a21-891a-5fdae7e11eb1-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.291839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291550 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02424f4-1f21-461e-9f87-dd21407f5292-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.291839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.291562 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84g7j\" (UniqueName: \"kubernetes.io/projected/c02424f4-1f21-461e-9f87-dd21407f5292-kube-api-access-84g7j\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.292193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.292161 4991 generic.go:356] "Generic (PLEG): container finished" podID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerID="d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26" exitCode=0 Oct 30 22:36:22.292301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.292280 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" event={"ID":"09394e12-bb87-4a21-891a-5fdae7e11eb1","Type":"ContainerDied","Data":"d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26"} Oct 30 22:36:22.292393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.292296 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" Oct 30 22:36:22.292393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.292358 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh" event={"ID":"09394e12-bb87-4a21-891a-5fdae7e11eb1","Type":"ContainerDied","Data":"65b96fad1b46bd28e705f1dc57239c94bfd206797445f524d7e48ba2e5b95fa4"} Oct 30 22:36:22.292393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.292391 4991 scope.go:122] "RemoveContainer" containerID="d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26" Oct 30 22:36:22.294564 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.294539 4991 generic.go:356] "Generic (PLEG): container finished" podID="c02424f4-1f21-461e-9f87-dd21407f5292" containerID="19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1" exitCode=0 Oct 30 22:36:22.295307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.295288 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" Oct 30 22:36:22.303156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.303131 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" event={"ID":"c02424f4-1f21-461e-9f87-dd21407f5292","Type":"ContainerDied","Data":"19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1"} Oct 30 22:36:22.303609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.303183 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt" event={"ID":"c02424f4-1f21-461e-9f87-dd21407f5292","Type":"ContainerDied","Data":"a68fddc74ff1be9fc340d9d9bf210335a232275d43bdb850ba9e9e66791e057b"} Oct 30 22:36:22.305912 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.305893 4991 scope.go:122] "RemoveContainer" containerID="d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26" Oct 30 22:36:22.306228 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:22.306202 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26\": container with ID starting with d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26 not found: ID does not exist" containerID="d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26" Oct 30 22:36:22.306361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.306232 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26"} err="failed to get container status \"d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26\": rpc error: code = NotFound desc = could not find container \"d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26\": container with ID starting with d2c8b306f7240f3f4ded7e022db3673bac34e028ba8416a1d610030dfa542a26 not found: ID does not exist" Oct 30 22:36:22.306361 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.306250 4991 scope.go:122] "RemoveContainer" containerID="19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1" Oct 30 22:36:22.316711 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.316689 4991 scope.go:122] "RemoveContainer" containerID="19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1" Oct 30 22:36:22.317001 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:22.316981 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1\": container with ID starting with 19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1 not found: ID does not exist" containerID="19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1" Oct 30 22:36:22.317053 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.317007 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1"} err="failed to get container status \"19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1\": rpc error: code = NotFound desc = could not find container \"19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1\": container with ID starting with 19f6b0851affa31c3ff40d8fb5704f26cdbf4bab486ded407a20f3becdea6bb1 not found: ID does not exist" Oct 30 22:36:22.451910 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.451879 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt"] Oct 30 22:36:22.457312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.457284 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-55pwt"] Oct 30 22:36:22.560423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.560358 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh"] Oct 30 22:36:22.565062 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:22.565038 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-q78lh"] Oct 30 22:36:23.874797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:23.874764 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" path="/var/lib/kubelet/pods/09394e12-bb87-4a21-891a-5fdae7e11eb1/volumes" Oct 30 22:36:23.875271 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:23.875257 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" path="/var/lib/kubelet/pods/c02424f4-1f21-461e-9f87-dd21407f5292/volumes" Oct 30 22:36:25.904902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.904841 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:36:25.937202 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937177 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-istio-token\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-istio-token\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937228 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-data\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937273 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-credential-socket\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937479 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937444 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8791caf8-3c35-4a37-9616-3098318e2f72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791caf8-3c35-4a37-9616-3098318e2f72-istiod-ca-cert\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937474 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-kube-api-access-shfd5\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-kube-api-access-shfd5\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937561 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8791caf8-3c35-4a37-9616-3098318e2f72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791caf8-3c35-4a37-9616-3098318e2f72-istio-podinfo\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937566 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-data" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:25.937705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937600 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-socket\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937605 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-credential-socket" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:25.937705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937650 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-certs\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937685 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-envoy\") pod \"8791caf8-3c35-4a37-9616-3098318e2f72\" (UID: \"8791caf8-3c35-4a37-9616-3098318e2f72\") " Oct 30 22:36:25.937897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937878 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-socket" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:25.937946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.937806 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8791caf8-3c35-4a37-9616-3098318e2f72-istiod-ca-cert" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:25.938749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.938624 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:25.938749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.938648 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:25.938749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.938664 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791caf8-3c35-4a37-9616-3098318e2f72-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:25.938749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.938678 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:25.938749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.938694 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:36:25.938749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.938705 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-certs" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:25.939946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.939925 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-istio-token" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:25.940049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.939994 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8791caf8-3c35-4a37-9616-3098318e2f72-istio-podinfo" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:25.940049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.940018 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-kube-api-access-shfd5" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "kube-api-access-shfd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:25.940146 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:25.940130 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-envoy" pod "8791caf8-3c35-4a37-9616-3098318e2f72" (UID: "8791caf8-3c35-4a37-9616-3098318e2f72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.039591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039557 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-socket\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.039751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039599 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-token\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-token\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.039751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039656 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-kube-api-access-4rxjn\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-kube-api-access-4rxjn\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.039878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039762 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-envoy\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.039878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039824 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-socket" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.039878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039862 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-credential-socket\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.040036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039910 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-data\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.040036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.039999 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-certs\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.040150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040045 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718a5205-8ad4-44a5-9529-cc6b5fee8737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718a5205-8ad4-44a5-9529-cc6b5fee8737-istiod-ca-cert\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.040150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040109 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-podinfo\") pod \"718a5205-8ad4-44a5-9529-cc6b5fee8737\" (UID: \"718a5205-8ad4-44a5-9529-cc6b5fee8737\") " Oct 30 22:36:26.040802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040107 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-credential-socket" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.040802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040245 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-data" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.040802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040425 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718a5205-8ad4-44a5-9529-cc6b5fee8737-istiod-ca-cert" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:26.040802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040768 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.040802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040796 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791caf8-3c35-4a37-9616-3098318e2f72-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.041156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040813 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.041156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040829 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.041156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040845 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfd5\" (UniqueName: \"kubernetes.io/projected/8791caf8-3c35-4a37-9616-3098318e2f72-kube-api-access-shfd5\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.041156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040863 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791caf8-3c35-4a37-9616-3098318e2f72-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.041156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.040651 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-certs" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.042226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.042199 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-envoy" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.042606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.042584 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-podinfo" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:26.042709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.042664 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-kube-api-access-4rxjn" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "kube-api-access-4rxjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:26.043484 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.043467 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-token" pod "718a5205-8ad4-44a5-9529-cc6b5fee8737" (UID: "718a5205-8ad4-44a5-9529-cc6b5fee8737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:26.142652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142619 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.142652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142648 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718a5205-8ad4-44a5-9529-cc6b5fee8737-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.142652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142658 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.142991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142668 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.142991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142680 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxjn\" (UniqueName: \"kubernetes.io/projected/718a5205-8ad4-44a5-9529-cc6b5fee8737-kube-api-access-4rxjn\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.142991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142689 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.142991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142697 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.142991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.142706 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718a5205-8ad4-44a5-9529-cc6b5fee8737-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.325604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.325568 4991 generic.go:356] "Generic (PLEG): container finished" podID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerID="54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec" exitCode=0 Oct 30 22:36:26.325804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.325637 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" Oct 30 22:36:26.325804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.325657 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" event={"ID":"718a5205-8ad4-44a5-9529-cc6b5fee8737","Type":"ContainerDied","Data":"54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec"} Oct 30 22:36:26.325804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.325729 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" event={"ID":"718a5205-8ad4-44a5-9529-cc6b5fee8737","Type":"ContainerDied","Data":"d3b97c6a120ea7e012249ecf2cd4c39d38eed94218552ad34385debfdf9587ef"} Oct 30 22:36:26.325804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.325760 4991 scope.go:122] "RemoveContainer" containerID="54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec" Oct 30 22:36:26.327304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.327279 4991 generic.go:356] "Generic (PLEG): container finished" podID="8791caf8-3c35-4a37-9616-3098318e2f72" containerID="3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c" exitCode=0 Oct 30 22:36:26.327451 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.327339 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" event={"ID":"8791caf8-3c35-4a37-9616-3098318e2f72","Type":"ContainerDied","Data":"3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c"} Oct 30 22:36:26.327451 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.327345 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" Oct 30 22:36:26.327451 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.327379 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv" event={"ID":"8791caf8-3c35-4a37-9616-3098318e2f72","Type":"ContainerDied","Data":"fdb1c8eff65c418275c434f96046a69228f63c7802ca52e84f6260821189dad5"} Oct 30 22:36:26.338649 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.338633 4991 scope.go:122] "RemoveContainer" containerID="54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec" Oct 30 22:36:26.338955 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:26.338922 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec\": container with ID starting with 54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec not found: ID does not exist" containerID="54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec" Oct 30 22:36:26.339055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.338967 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec"} err="failed to get container status \"54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec\": rpc error: code = NotFound desc = could not find container \"54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec\": container with ID starting with 54089219a104953afb79445028181fbaff8816431fc75cc3db63e7cec62afeec not found: ID does not exist" Oct 30 22:36:26.339055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.338986 4991 scope.go:122] "RemoveContainer" containerID="3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c" Oct 30 22:36:26.352350 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.352303 4991 scope.go:122] "RemoveContainer" containerID="3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c" Oct 30 22:36:26.352769 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:26.352748 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c\": container with ID starting with 3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c not found: ID does not exist" containerID="3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c" Oct 30 22:36:26.352868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.352779 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c"} err="failed to get container status \"3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c\": rpc error: code = NotFound desc = could not find container \"3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c\": container with ID starting with 3923afbec4550c48c7e2657bc70c6947a73460330c9be952c51135267bf7069c not found: ID does not exist" Oct 30 22:36:26.482799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.482768 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv"] Oct 30 22:36:26.492473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.492446 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wndjv"] Oct 30 22:36:26.595658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.595586 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb"] Oct 30 22:36:26.599805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.599779 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb"] Oct 30 22:36:26.875522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.875437 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:36:26.875522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:26.875509 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-44mgb" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:36:27.876768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:27.876734 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" path="/var/lib/kubelet/pods/718a5205-8ad4-44a5-9529-cc6b5fee8737/volumes" Oct 30 22:36:27.877224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:27.877210 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" path="/var/lib/kubelet/pods/8791caf8-3c35-4a37-9616-3098318e2f72/volumes" Oct 30 22:36:29.461727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:29.461688 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:29.476296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:29.476271 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:30.371617 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:36:30.371525 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:36:47.910596 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:36:47.910558 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:37:01.708396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:01.708360 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt"] Oct 30 22:37:01.708911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:01.708565 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" containerID="cri-o://88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2" gracePeriod=30 Oct 30 22:37:03.801959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:03.801920 4991 patch_prober.go:29] interesting pod/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:37:03.802432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:03.801986 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:37:04.032956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.032917 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk"] Oct 30 22:37:04.033860 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033838 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" Oct 30 22:37:04.033925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033864 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" Oct 30 22:37:04.033925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033897 4991 cpu_manager.go:414] "containerMap: removing container" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" Oct 30 22:37:04.033925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033905 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" Oct 30 22:37:04.034027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033925 4991 cpu_manager.go:414] "containerMap: removing container" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" Oct 30 22:37:04.034027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033934 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" Oct 30 22:37:04.034027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033944 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" Oct 30 22:37:04.034027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033952 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" Oct 30 22:37:04.034027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033973 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" Oct 30 22:37:04.034027 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.033981 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" Oct 30 22:37:04.034201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.034110 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02424f4-1f21-461e-9f87-dd21407f5292" containerName="istio-proxy" Oct 30 22:37:04.034201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.034127 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0ebfa3-73bf-4704-9c0d-65b0d9e0267c" containerName="istio-proxy" Oct 30 22:37:04.034201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.034139 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="09394e12-bb87-4a21-891a-5fdae7e11eb1" containerName="istio-proxy" Oct 30 22:37:04.034201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.034149 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="718a5205-8ad4-44a5-9529-cc6b5fee8737" containerName="istio-proxy" Oct 30 22:37:04.034201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.034158 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8791caf8-3c35-4a37-9616-3098318e2f72" containerName="istio-proxy" Oct 30 22:37:04.038814 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.038793 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.091598 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.091521 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk"] Oct 30 22:37:04.127735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.127676 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tls-certs\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.127895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.127794 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t789m\" (UniqueName: \"kubernetes.io/projected/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-kube-api-access-t789m\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.127895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.127833 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-home\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.127895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.127864 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tmp-dir\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.128007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.127920 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-model-cache\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.128007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.127995 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-dshm\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.133535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.133504 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-37e79b\"/\"auth-test-model-37e79b-kserve-self-signed-certs\"" Oct 30 22:37:04.133701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.133611 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-37e79b\"/\"kube-root-ca.crt\"" Oct 30 22:37:04.134857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.134837 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-37e79b\"/\"openshift-service-ca.crt\"" Oct 30 22:37:04.134988 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.134854 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-37e79b\"/\"default-dockercfg-wbkgw\"" Oct 30 22:37:04.230218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.230178 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-model-cache\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.230615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.230562 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-dshm\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.230901 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.230881 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tls-certs\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.231189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.231164 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t789m\" (UniqueName: \"kubernetes.io/projected/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-kube-api-access-t789m\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.231286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.231213 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-home\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.231286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.231265 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tmp-dir\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.231526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.230644 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-model-cache\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.231677 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.231656 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tmp-dir\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.231841 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.231730 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-home\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.234125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.234083 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tls-certs\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.247719 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.247698 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-dshm\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.316655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.316622 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t789m\" (UniqueName: \"kubernetes.io/projected/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-kube-api-access-t789m\") pod \"auth-test-model-37e79b-kserve-5c5c957469-tj9tk\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.354706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.354634 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:04.735594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:04.735565 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk"] Oct 30 22:37:04.746974 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:37:04.746946 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod288c14ac_8b95_4a43_b368_e11e9a5cbaa1.slice/crio-a66fe451f32d133eead1ddb86addcb9a0a43f896e62d7524e54ad8286c08f6ef WatchSource:0}: Error finding container a66fe451f32d133eead1ddb86addcb9a0a43f896e62d7524e54ad8286c08f6ef: Status 404 returned error can't find the container with id a66fe451f32d133eead1ddb86addcb9a0a43f896e62d7524e54ad8286c08f6ef Oct 30 22:37:05.625272 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:05.625231 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" event={"ID":"288c14ac-8b95-4a43-b368-e11e9a5cbaa1","Type":"ContainerStarted","Data":"b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3"} Oct 30 22:37:05.625763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:05.625282 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" event={"ID":"288c14ac-8b95-4a43-b368-e11e9a5cbaa1","Type":"ContainerStarted","Data":"a66fe451f32d133eead1ddb86addcb9a0a43f896e62d7524e54ad8286c08f6ef"} Oct 30 22:37:05.736777 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:05.736711 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" podStartSLOduration=2.736692772 podStartE2EDuration="2.736692772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:05.735491297 +0000 UTC m=+3046.034901195" watchObservedRunningTime="2026-10-30 22:37:05.736692772 +0000 UTC m=+3046.036102654" Oct 30 22:37:06.355212 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:06.355183 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:06.987974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:06.987950 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:37:07.099464 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099432 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-envoy\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.099596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099512 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e698f07-79f2-451e-8777-e73f923096c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e698f07-79f2-451e-8777-e73f923096c1-istio-podinfo\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.099596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099568 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-credential-socket\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.099726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099615 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-socket\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.099726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099688 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e698f07-79f2-451e-8777-e73f923096c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e698f07-79f2-451e-8777-e73f923096c1-istiod-ca-cert\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.099836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099729 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-kube-api-access-57pfg\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-kube-api-access-57pfg\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.099932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099883 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-data\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.100058 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099888 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-credential-socket" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:07.100058 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099901 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-socket" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:07.100058 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.099940 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-certs\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.100224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100071 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-istio-token\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-istio-token\") pod \"9e698f07-79f2-451e-8777-e73f923096c1\" (UID: \"9e698f07-79f2-451e-8777-e73f923096c1\") " Oct 30 22:37:07.100224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100109 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-data" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:07.100224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100146 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-certs" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:07.100224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100195 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e698f07-79f2-451e-8777-e73f923096c1-istiod-ca-cert" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:07.100539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100519 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.100585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100545 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.100585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100558 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e698f07-79f2-451e-8777-e73f923096c1-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.100585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100571 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.100701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.100584 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.101978 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.101953 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-kube-api-access-57pfg" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "kube-api-access-57pfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:07.102185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.102157 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-istio-token" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:07.102185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.102164 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-envoy" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:07.102364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.102225 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e698f07-79f2-451e-8777-e73f923096c1-istio-podinfo" pod "9e698f07-79f2-451e-8777-e73f923096c1" (UID: "9e698f07-79f2-451e-8777-e73f923096c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:37:07.201680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.201643 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e698f07-79f2-451e-8777-e73f923096c1-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.201680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.201675 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pfg\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-kube-api-access-57pfg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.201875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.201690 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e698f07-79f2-451e-8777-e73f923096c1-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.201875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.201706 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e698f07-79f2-451e-8777-e73f923096c1-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:07.643389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.643353 4991 generic.go:356] "Generic (PLEG): container finished" podID="9e698f07-79f2-451e-8777-e73f923096c1" containerID="88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2" exitCode=0 Oct 30 22:37:07.643547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.643416 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" Oct 30 22:37:07.643547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.643432 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" event={"ID":"9e698f07-79f2-451e-8777-e73f923096c1","Type":"ContainerDied","Data":"88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2"} Oct 30 22:37:07.643547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.643489 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt" event={"ID":"9e698f07-79f2-451e-8777-e73f923096c1","Type":"ContainerDied","Data":"37d528abe7c120b285dd9a129fc2d487c5d8fe695ba43a94c818bd16d7f21aa9"} Oct 30 22:37:07.643547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.643516 4991 scope.go:122] "RemoveContainer" containerID="88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2" Oct 30 22:37:07.655365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.655304 4991 scope.go:122] "RemoveContainer" containerID="15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422" Oct 30 22:37:07.666916 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.666893 4991 scope.go:122] "RemoveContainer" containerID="88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2" Oct 30 22:37:07.667187 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:37:07.667168 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2\": container with ID starting with 88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2 not found: ID does not exist" containerID="88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2" Oct 30 22:37:07.667265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.667198 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2"} err="failed to get container status \"88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2\": rpc error: code = NotFound desc = could not find container \"88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2\": container with ID starting with 88480005faa5d8d04ccd6c645dee36210cd9a31621841120e54f066779bf37a2 not found: ID does not exist" Oct 30 22:37:07.667265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.667217 4991 scope.go:122] "RemoveContainer" containerID="15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422" Oct 30 22:37:07.667492 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:37:07.667474 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422\": container with ID starting with 15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422 not found: ID does not exist" containerID="15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422" Oct 30 22:37:07.667556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.667497 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422"} err="failed to get container status \"15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422\": rpc error: code = NotFound desc = could not find container \"15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422\": container with ID starting with 15ebb8d31c7b1c7415d33ece94c29a8afc91efa4694bd9aeb17214a05667e422 not found: ID does not exist" Oct 30 22:37:07.802568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.802528 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt"] Oct 30 22:37:07.849335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.849303 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1875b4-openshift-default-54b87966f-kgbbt"] Oct 30 22:37:07.875825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:07.875792 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e698f07-79f2-451e-8777-e73f923096c1" path="/var/lib/kubelet/pods/9e698f07-79f2-451e-8777-e73f923096c1/volumes" Oct 30 22:37:14.173655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:14.173617 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9"] Oct 30 22:37:14.293201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:14.293168 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr"] Oct 30 22:37:14.355490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:14.355456 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:14.372837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:14.372813 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:14.708939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:14.708908 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:37:17.911022 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:37:17.910987 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:37:19.175004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.174944 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" podUID="94973040-d19d-4a9a-8b45-b8cdd3bfd176" containerName="payload-processing" containerID="cri-o://792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2" gracePeriod=25 Oct 30 22:37:19.294411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.294376 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" podUID="72224bc6-33bb-43a2-bd01-df0109a97984" containerName="payload-pre-processing" containerID="cri-o://45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89" gracePeriod=25 Oct 30 22:37:19.463900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.463870 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:37:19.572430 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.572401 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94973040-d19d-4a9a-8b45-b8cdd3bfd176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94973040-d19d-4a9a-8b45-b8cdd3bfd176-plugins-config-volume\") pod \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " Oct 30 22:37:19.572556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.572500 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94973040-d19d-4a9a-8b45-b8cdd3bfd176-kube-api-access-xdzrs\" (UniqueName: \"kubernetes.io/projected/94973040-d19d-4a9a-8b45-b8cdd3bfd176-kube-api-access-xdzrs\") pod \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\" (UID: \"94973040-d19d-4a9a-8b45-b8cdd3bfd176\") " Oct 30 22:37:19.572793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.572770 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94973040-d19d-4a9a-8b45-b8cdd3bfd176-plugins-config-volume" pod "94973040-d19d-4a9a-8b45-b8cdd3bfd176" (UID: "94973040-d19d-4a9a-8b45-b8cdd3bfd176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:19.573056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.573035 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94973040-d19d-4a9a-8b45-b8cdd3bfd176-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:19.574623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.574599 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94973040-d19d-4a9a-8b45-b8cdd3bfd176-kube-api-access-xdzrs" pod "94973040-d19d-4a9a-8b45-b8cdd3bfd176" (UID: "94973040-d19d-4a9a-8b45-b8cdd3bfd176"). InnerVolumeSpecName "kube-api-access-xdzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:19.584385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.580671 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:37:19.673834 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.673798 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72224bc6-33bb-43a2-bd01-df0109a97984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72224bc6-33bb-43a2-bd01-df0109a97984-plugins-config-volume\") pod \"72224bc6-33bb-43a2-bd01-df0109a97984\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " Oct 30 22:37:19.674006 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.673898 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72224bc6-33bb-43a2-bd01-df0109a97984-kube-api-access-zdx6t\" (UniqueName: \"kubernetes.io/projected/72224bc6-33bb-43a2-bd01-df0109a97984-kube-api-access-zdx6t\") pod \"72224bc6-33bb-43a2-bd01-df0109a97984\" (UID: \"72224bc6-33bb-43a2-bd01-df0109a97984\") " Oct 30 22:37:19.674222 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.674198 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72224bc6-33bb-43a2-bd01-df0109a97984-plugins-config-volume" pod "72224bc6-33bb-43a2-bd01-df0109a97984" (UID: "72224bc6-33bb-43a2-bd01-df0109a97984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:19.674366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.674347 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72224bc6-33bb-43a2-bd01-df0109a97984-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:19.674410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.674373 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdzrs\" (UniqueName: \"kubernetes.io/projected/94973040-d19d-4a9a-8b45-b8cdd3bfd176-kube-api-access-xdzrs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:19.675918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.675882 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72224bc6-33bb-43a2-bd01-df0109a97984-kube-api-access-zdx6t" pod "72224bc6-33bb-43a2-bd01-df0109a97984" (UID: "72224bc6-33bb-43a2-bd01-df0109a97984"). InnerVolumeSpecName "kube-api-access-zdx6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:19.779198 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.779163 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdx6t\" (UniqueName: \"kubernetes.io/projected/72224bc6-33bb-43a2-bd01-df0109a97984-kube-api-access-zdx6t\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:19.782334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.782293 4991 generic.go:356] "Generic (PLEG): container finished" podID="94973040-d19d-4a9a-8b45-b8cdd3bfd176" containerID="792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2" exitCode=0 Oct 30 22:37:19.782496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.782353 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" event={"ID":"94973040-d19d-4a9a-8b45-b8cdd3bfd176","Type":"ContainerDied","Data":"792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2"} Oct 30 22:37:19.782496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.782370 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" Oct 30 22:37:19.782496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.782449 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9" event={"ID":"94973040-d19d-4a9a-8b45-b8cdd3bfd176","Type":"ContainerDied","Data":"da00492e0f49686b258bd94e91b9afc9a58c6701efedd86bc01dab1a9b4bcf83"} Oct 30 22:37:19.782496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.782488 4991 scope.go:122] "RemoveContainer" containerID="792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2" Oct 30 22:37:19.783960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.783932 4991 generic.go:356] "Generic (PLEG): container finished" podID="72224bc6-33bb-43a2-bd01-df0109a97984" containerID="45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89" exitCode=0 Oct 30 22:37:19.784067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.783979 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" Oct 30 22:37:19.784067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.783976 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" event={"ID":"72224bc6-33bb-43a2-bd01-df0109a97984","Type":"ContainerDied","Data":"45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89"} Oct 30 22:37:19.784180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.784111 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr" event={"ID":"72224bc6-33bb-43a2-bd01-df0109a97984","Type":"ContainerDied","Data":"5391eb784895399f9a4427f30479d0e328c10965d4b368abe7ad2ebf97e6c208"} Oct 30 22:37:19.793240 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.793225 4991 scope.go:122] "RemoveContainer" containerID="792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2" Oct 30 22:37:19.793593 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:37:19.793550 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2\": container with ID starting with 792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2 not found: ID does not exist" containerID="792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2" Oct 30 22:37:19.793697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.793608 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2"} err="failed to get container status \"792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2\": rpc error: code = NotFound desc = could not find container \"792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2\": container with ID starting with 792bdce0b6d4816f672e74061e694273f74a1fa06c645e177776d6230a63b4a2 not found: ID does not exist" Oct 30 22:37:19.793697 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.793643 4991 scope.go:122] "RemoveContainer" containerID="45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89" Oct 30 22:37:19.802876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.802855 4991 scope.go:122] "RemoveContainer" containerID="45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89" Oct 30 22:37:19.803124 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:37:19.803105 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89\": container with ID starting with 45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89 not found: ID does not exist" containerID="45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89" Oct 30 22:37:19.803178 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.803131 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89"} err="failed to get container status \"45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89\": rpc error: code = NotFound desc = could not find container \"45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89\": container with ID starting with 45d1f308ce1479943d9b1baeb3a5c5c94df9a98b71c274dec996d5b8862edc89 not found: ID does not exist" Oct 30 22:37:19.939902 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.939865 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9"] Oct 30 22:37:19.947073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:19.947044 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-28f124-58d9dffc6c-rhhk9"] Oct 30 22:37:20.049822 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:20.049738 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr"] Oct 30 22:37:20.056581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:20.056544 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-28f124-5765fcc8c-94wfr"] Oct 30 22:37:21.875815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:21.875785 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72224bc6-33bb-43a2-bd01-df0109a97984" path="/var/lib/kubelet/pods/72224bc6-33bb-43a2-bd01-df0109a97984/volumes" Oct 30 22:37:21.876833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:37:21.876808 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94973040-d19d-4a9a-8b45-b8cdd3bfd176" path="/var/lib/kubelet/pods/94973040-d19d-4a9a-8b45-b8cdd3bfd176/volumes" Oct 30 22:37:47.911941 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:37:47.911904 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:38:13.697055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:13.697022 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt"] Oct 30 22:38:13.697810 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:13.697243 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" podUID="74d7f3e8-b372-4e10-96ca-9099a500c474" containerName="maas-api" containerID="cri-o://bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b" gracePeriod=30 Oct 30 22:38:13.985097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:13.985070 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:38:14.112450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.112408 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d7f3e8-b372-4e10-96ca-9099a500c474-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d7f3e8-b372-4e10-96ca-9099a500c474-openshift-service-ca\") pod \"74d7f3e8-b372-4e10-96ca-9099a500c474\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " Oct 30 22:38:14.112655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.112512 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls\") pod \"74d7f3e8-b372-4e10-96ca-9099a500c474\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " Oct 30 22:38:14.112655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.112605 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d7f3e8-b372-4e10-96ca-9099a500c474-kube-api-access-flmvp\" (UniqueName: \"kubernetes.io/projected/74d7f3e8-b372-4e10-96ca-9099a500c474-kube-api-access-flmvp\") pod \"74d7f3e8-b372-4e10-96ca-9099a500c474\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " Oct 30 22:38:14.112803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.112766 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d7f3e8-b372-4e10-96ca-9099a500c474-openshift-service-ca" pod "74d7f3e8-b372-4e10-96ca-9099a500c474" (UID: "74d7f3e8-b372-4e10-96ca-9099a500c474"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:14.112803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.112794 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-metrics-certs\") pod \"74d7f3e8-b372-4e10-96ca-9099a500c474\" (UID: \"74d7f3e8-b372-4e10-96ca-9099a500c474\") " Oct 30 22:38:14.113382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.113359 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d7f3e8-b372-4e10-96ca-9099a500c474-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.115719 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.115682 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls" pod "74d7f3e8-b372-4e10-96ca-9099a500c474" (UID: "74d7f3e8-b372-4e10-96ca-9099a500c474"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:14.115993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.115962 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-metrics-certs" pod "74d7f3e8-b372-4e10-96ca-9099a500c474" (UID: "74d7f3e8-b372-4e10-96ca-9099a500c474"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:14.133652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.133617 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d7f3e8-b372-4e10-96ca-9099a500c474-kube-api-access-flmvp" pod "74d7f3e8-b372-4e10-96ca-9099a500c474" (UID: "74d7f3e8-b372-4e10-96ca-9099a500c474"). InnerVolumeSpecName "kube-api-access-flmvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:14.163797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.163762 4991 generic.go:356] "Generic (PLEG): container finished" podID="74d7f3e8-b372-4e10-96ca-9099a500c474" containerID="bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b" exitCode=0 Oct 30 22:38:14.163939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.163805 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" Oct 30 22:38:14.163939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.163837 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" event={"ID":"74d7f3e8-b372-4e10-96ca-9099a500c474","Type":"ContainerDied","Data":"bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b"} Oct 30 22:38:14.163939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.163890 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt" event={"ID":"74d7f3e8-b372-4e10-96ca-9099a500c474","Type":"ContainerDied","Data":"be2ca8fd8d4fa12e8921ae218e7eddf7d207231938d4d4943faa84f5524d291f"} Oct 30 22:38:14.163939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.163912 4991 scope.go:122] "RemoveContainer" containerID="bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b" Oct 30 22:38:14.175995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.175978 4991 scope.go:122] "RemoveContainer" containerID="bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b" Oct 30 22:38:14.176261 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:38:14.176241 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b\": container with ID starting with bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b not found: ID does not exist" containerID="bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b" Oct 30 22:38:14.176308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.176273 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b"} err="failed to get container status \"bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b\": rpc error: code = NotFound desc = could not find container \"bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b\": container with ID starting with bbebd8818df96e1f56a4dae2907216173494ffdc8bbb895399d54e5618eaa45b not found: ID does not exist" Oct 30 22:38:14.214497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.214443 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.214497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.214470 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d7f3e8-b372-4e10-96ca-9099a500c474-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.214497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.214481 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flmvp\" (UniqueName: \"kubernetes.io/projected/74d7f3e8-b372-4e10-96ca-9099a500c474-kube-api-access-flmvp\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.337009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.336974 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt"] Oct 30 22:38:14.340129 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:14.340099 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-28f124-77f6788cc4-dgvbt"] Oct 30 22:38:15.130273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.130234 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl"] Oct 30 22:38:15.131071 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131047 4991 cpu_manager.go:414] "containerMap: removing container" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" Oct 30 22:38:15.131123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131075 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" Oct 30 22:38:15.131123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131103 4991 cpu_manager.go:414] "containerMap: removing container" podUID="74d7f3e8-b372-4e10-96ca-9099a500c474" containerName="maas-api" Oct 30 22:38:15.131123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131110 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d7f3e8-b372-4e10-96ca-9099a500c474" containerName="maas-api" Oct 30 22:38:15.131220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131145 4991 cpu_manager.go:414] "containerMap: removing container" podUID="72224bc6-33bb-43a2-bd01-df0109a97984" containerName="payload-pre-processing" Oct 30 22:38:15.131220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131154 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72224bc6-33bb-43a2-bd01-df0109a97984" containerName="payload-pre-processing" Oct 30 22:38:15.131220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131176 4991 cpu_manager.go:414] "containerMap: removing container" podUID="94973040-d19d-4a9a-8b45-b8cdd3bfd176" containerName="payload-processing" Oct 30 22:38:15.131220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131183 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94973040-d19d-4a9a-8b45-b8cdd3bfd176" containerName="payload-processing" Oct 30 22:38:15.131365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131354 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="94973040-d19d-4a9a-8b45-b8cdd3bfd176" containerName="payload-processing" Oct 30 22:38:15.131401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131373 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" Oct 30 22:38:15.131401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131384 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" Oct 30 22:38:15.131401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131394 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d7f3e8-b372-4e10-96ca-9099a500c474" containerName="maas-api" Oct 30 22:38:15.131492 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.131409 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="72224bc6-33bb-43a2-bd01-df0109a97984" containerName="payload-pre-processing" Oct 30 22:38:15.136738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.136715 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.230652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.230621 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-435923\"/\"openshift-service-ca.crt\"" Oct 30 22:38:15.231677 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.231590 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-dshm\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.231929 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.231910 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-home\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.232302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.232277 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tmp-dir\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.232513 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.232494 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fb6\" (UniqueName: \"kubernetes.io/projected/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-kube-api-access-t7fb6\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.232712 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.232696 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-model-cache\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.232956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.232936 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tls-certs\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.236147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.236125 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-435923\"/\"kube-root-ca.crt\"" Oct 30 22:38:15.236333 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.236301 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-435923\"/\"default-dockercfg-4stbf\"" Oct 30 22:38:15.236461 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.236440 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-435923\"/\"auth-test-model-435923-kserve-self-signed-certs\"" Oct 30 22:38:15.240962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.240942 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl"] Oct 30 22:38:15.334135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334096 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-home\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334276 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tmp-dir\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334349 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fb6\" (UniqueName: \"kubernetes.io/projected/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-kube-api-access-t7fb6\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334423 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-model-cache\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334501 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tls-certs\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334550 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-dshm\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334579 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-home\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334648 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tmp-dir\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.334990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.334969 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-model-cache\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.336945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.336917 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-dshm\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.337442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.337422 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tls-certs\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.397531 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.397469 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fb6\" (UniqueName: \"kubernetes.io/projected/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-kube-api-access-t7fb6\") pod \"auth-test-model-435923-kserve-5dcd6cdb89-8kkhl\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.456584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.456556 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:15.876221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.876182 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d7f3e8-b372-4e10-96ca-9099a500c474" path="/var/lib/kubelet/pods/74d7f3e8-b372-4e10-96ca-9099a500c474/volumes" Oct 30 22:38:15.876562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:15.876547 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl"] Oct 30 22:38:15.881049 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:38:15.881023 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07c88d2f_f7e8_480a_a4ae_b7b3f7a83977.slice/crio-ab187c7998ef0c81243c9227bb8b2e19dbf0812933b25db04f78f8da15856027 WatchSource:0}: Error finding container ab187c7998ef0c81243c9227bb8b2e19dbf0812933b25db04f78f8da15856027: Status 404 returned error can't find the container with id ab187c7998ef0c81243c9227bb8b2e19dbf0812933b25db04f78f8da15856027 Oct 30 22:38:16.189721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:16.189633 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" event={"ID":"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977","Type":"ContainerStarted","Data":"b99a20acd8a72617e5cca47cd1ffbbb554cf8e378a03581f6d29ab51c8d5c6fd"} Oct 30 22:38:16.189721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:16.189686 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" event={"ID":"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977","Type":"ContainerStarted","Data":"ab187c7998ef0c81243c9227bb8b2e19dbf0812933b25db04f78f8da15856027"} Oct 30 22:38:16.305790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:16.305735 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" podStartSLOduration=1.305721659 podStartE2EDuration="1.305721659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:16.303846548 +0000 UTC m=+3116.603256452" watchObservedRunningTime="2026-10-30 22:38:16.305721659 +0000 UTC m=+3116.605131539" Oct 30 22:38:16.457511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:16.457438 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:17.625334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:17.625295 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch"] Oct 30 22:38:17.625906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:17.625516 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" containerID="cri-o://9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835" gracePeriod=30 Oct 30 22:38:17.911969 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:38:17.911889 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:38:18.631424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.631393 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f"] Oct 30 22:38:18.631960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.631944 4991 cpu_manager.go:414] "containerMap: removing container" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" Oct 30 22:38:18.632010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.631962 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e698f07-79f2-451e-8777-e73f923096c1" containerName="istio-proxy" Oct 30 22:38:18.635973 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.635954 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.678925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.678890 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-model-cache\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.679069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.679017 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-dshm\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.679156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.679136 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e88a477-7c03-4205-afe5-90418cfd3fd2-tls-certs\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.679237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.679220 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzwg\" (UniqueName: \"kubernetes.io/projected/2e88a477-7c03-4205-afe5-90418cfd3fd2-kube-api-access-sfzwg\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.679300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.679284 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-home\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.679366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.679315 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-tmp-dir\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.709852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.709825 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f"] Oct 30 22:38:18.732990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.732958 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3b392d\"/\"kube-root-ca.crt\"" Oct 30 22:38:18.732990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.732970 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3b392d\"/\"openshift-service-ca.crt\"" Oct 30 22:38:18.733194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.733025 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3b392d\"/\"default-dockercfg-2g8c9\"" Oct 30 22:38:18.733194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.733104 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3b392d\"/\"auth-test-model-3b392d-kserve-self-signed-certs\"" Oct 30 22:38:18.780302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.780269 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-model-cache\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.780302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.780416 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-dshm\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.780302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.780485 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e88a477-7c03-4205-afe5-90418cfd3fd2-tls-certs\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.780302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.780556 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzwg\" (UniqueName: \"kubernetes.io/projected/2e88a477-7c03-4205-afe5-90418cfd3fd2-kube-api-access-sfzwg\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.780302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.780620 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-home\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.780302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.780646 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-tmp-dir\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.780302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.782733 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-model-cache\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.784426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.783736 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-home\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.794816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.794778 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-tmp-dir\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.801365 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.800006 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-dshm\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.810307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.809887 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e88a477-7c03-4205-afe5-90418cfd3fd2-tls-certs\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.841598 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.841555 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzwg\" (UniqueName: \"kubernetes.io/projected/2e88a477-7c03-4205-afe5-90418cfd3fd2-kube-api-access-sfzwg\") pod \"auth-test-model-3b392d-kserve-69db4758bf-hx62f\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:18.952293 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:18.952219 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:19.332545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:19.332509 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f"] Oct 30 22:38:19.339354 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:38:19.339307 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e88a477_7c03_4205_afe5_90418cfd3fd2.slice/crio-4030eec5518febb5f612adc611633e232f29f5da5e7b8a11b005e4eb6378c53b WatchSource:0}: Error finding container 4030eec5518febb5f612adc611633e232f29f5da5e7b8a11b005e4eb6378c53b: Status 404 returned error can't find the container with id 4030eec5518febb5f612adc611633e232f29f5da5e7b8a11b005e4eb6378c53b Oct 30 22:38:20.222131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:20.222094 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" event={"ID":"2e88a477-7c03-4205-afe5-90418cfd3fd2","Type":"ContainerStarted","Data":"422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23"} Oct 30 22:38:20.222652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:20.222145 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" event={"ID":"2e88a477-7c03-4205-afe5-90418cfd3fd2","Type":"ContainerStarted","Data":"4030eec5518febb5f612adc611633e232f29f5da5e7b8a11b005e4eb6378c53b"} Oct 30 22:38:20.333987 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:20.333938 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" podStartSLOduration=2.333925251 podStartE2EDuration="2.333925251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:20.33218844 +0000 UTC m=+3120.631598322" watchObservedRunningTime="2026-10-30 22:38:20.333925251 +0000 UTC m=+3120.633335206" Oct 30 22:38:20.959344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:20.952424 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:22.904990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.904938 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:38:22.944294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944265 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-certs\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944299 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-token\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-token\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944413 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-envoy\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944444 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istiod-ca-cert\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944472 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-credential-socket\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944522 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-podinfo\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944572 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-kube-api-access-5xxs2\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-kube-api-access-5xxs2\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944571 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-certs" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:22.944849 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944795 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-credential-socket" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:22.944849 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944840 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-data\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.944966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944878 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istiod-ca-cert" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:22.944966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.944889 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-socket\") pod \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\" (UID: \"cde3b88b-30ce-4dc2-9a43-00145f5d964a\") " Oct 30 22:38:22.945113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.945089 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-data" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:22.945267 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.945242 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-socket" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:22.945803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.945555 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:22.945803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.945581 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:22.945803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.945598 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:22.945803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.945612 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:22.945803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.945626 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:22.946772 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.946749 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-token" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:22.946772 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.946762 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-envoy" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:22.946961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.946752 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-podinfo" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:38:22.947007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:22.946987 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-kube-api-access-5xxs2" pod "cde3b88b-30ce-4dc2-9a43-00145f5d964a" (UID: "cde3b88b-30ce-4dc2-9a43-00145f5d964a"). InnerVolumeSpecName "kube-api-access-5xxs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:23.047486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.047446 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:23.047486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.047484 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:23.047665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.047500 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxs2\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-kube-api-access-5xxs2\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:23.047665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.047518 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde3b88b-30ce-4dc2-9a43-00145f5d964a-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:23.245220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.245182 4991 generic.go:356] "Generic (PLEG): container finished" podID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerID="9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835" exitCode=0 Oct 30 22:38:23.245404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.245242 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" Oct 30 22:38:23.245404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.245269 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" event={"ID":"cde3b88b-30ce-4dc2-9a43-00145f5d964a","Type":"ContainerDied","Data":"9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835"} Oct 30 22:38:23.245404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.245374 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch" event={"ID":"cde3b88b-30ce-4dc2-9a43-00145f5d964a","Type":"ContainerDied","Data":"e3c1d7750f5ca1594fc19ead7a7eeb704563ab4febdce77d210a6fa63a153d7c"} Oct 30 22:38:23.245585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.245407 4991 scope.go:122] "RemoveContainer" containerID="9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835" Oct 30 22:38:23.257446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.256949 4991 scope.go:122] "RemoveContainer" containerID="9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835" Oct 30 22:38:23.257446 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:38:23.257240 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835\": container with ID starting with 9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835 not found: ID does not exist" containerID="9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835" Oct 30 22:38:23.257446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.257265 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835"} err="failed to get container status \"9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835\": rpc error: code = NotFound desc = could not find container \"9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835\": container with ID starting with 9c1bdd2db633a872a842c68b7c773d611b1f4f02232820b13d1214974b169835 not found: ID does not exist" Oct 30 22:38:23.405563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.405533 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch"] Oct 30 22:38:23.410398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.410370 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-cdpch"] Oct 30 22:38:23.878901 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:23.878871 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" path="/var/lib/kubelet/pods/cde3b88b-30ce-4dc2-9a43-00145f5d964a/volumes" Oct 30 22:38:25.456833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:25.456798 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:25.469393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:25.469367 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:26.279932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:26.279904 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:38:28.952666 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:28.952604 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:28.966125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:28.966095 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:29.305732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:29.305695 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:38:43.110417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.110370 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t"] Oct 30 22:38:43.110931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.110914 4991 cpu_manager.go:414] "containerMap: removing container" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" Oct 30 22:38:43.110931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.110930 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" Oct 30 22:38:43.111066 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.111047 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde3b88b-30ce-4dc2-9a43-00145f5d964a" containerName="istio-proxy" Oct 30 22:38:43.115829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.115803 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.167765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.167730 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-cbe3a9-openshift-default-dockercfg-4z9tw\"" Oct 30 22:38:43.219796 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.219761 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t"] Oct 30 22:38:43.220543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.220515 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-credential-socket\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.220813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.220790 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-istio-token\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.221238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.221216 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67663820-6166-4317-a27d-83bd8d2e7843-istio-podinfo\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.221441 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.221414 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67663820-6166-4317-a27d-83bd8d2e7843-istiod-ca-cert\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.221561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.221446 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gfds\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-kube-api-access-6gfds\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.221561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.221497 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-socket\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.221561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.221529 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-certs\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.221774 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.221682 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-envoy\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.222241 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.222199 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-data\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.328909 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.328872 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-credential-socket\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.329163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.329138 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-istio-token\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.329359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.329306 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-credential-socket\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.329471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.329349 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67663820-6166-4317-a27d-83bd8d2e7843-istio-podinfo\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.329590 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.329574 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67663820-6166-4317-a27d-83bd8d2e7843-istiod-ca-cert\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.329710 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.329692 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gfds\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-kube-api-access-6gfds\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.329993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.329968 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-socket\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.330095 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.330017 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-certs\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.330095 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.330059 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-envoy\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.330222 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.330110 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-data\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.330573 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.330547 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67663820-6166-4317-a27d-83bd8d2e7843-istiod-ca-cert\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.330676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.330616 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-data\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.330816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.330794 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-socket\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.331014 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.330995 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-certs\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.333033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.333011 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67663820-6166-4317-a27d-83bd8d2e7843-istio-podinfo\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.334083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.334057 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-envoy\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.388150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.388046 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gfds\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-kube-api-access-6gfds\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.394752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.394724 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-istio-token\") pod \"e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.435355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.435273 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:43.899076 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.899042 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t"] Oct 30 22:38:43.905762 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:38:43.905733 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67663820_6166_4317_a27d_83bd8d2e7843.slice/crio-f6d6f57f2641209a3fda7b48583c51651cd41986b8089eaa2fc754e8920a8884 WatchSource:0}: Error finding container f6d6f57f2641209a3fda7b48583c51651cd41986b8089eaa2fc754e8920a8884: Status 404 returned error can't find the container with id f6d6f57f2641209a3fda7b48583c51651cd41986b8089eaa2fc754e8920a8884 Oct 30 22:38:43.909025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.908982 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:38:43.909148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.909075 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:38:43.909148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:43.909121 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:38:44.406626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:44.406592 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" event={"ID":"67663820-6166-4317-a27d-83bd8d2e7843","Type":"ContainerStarted","Data":"aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7"} Oct 30 22:38:44.407118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:44.406652 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" event={"ID":"67663820-6166-4317-a27d-83bd8d2e7843","Type":"ContainerStarted","Data":"f6d6f57f2641209a3fda7b48583c51651cd41986b8089eaa2fc754e8920a8884"} Oct 30 22:38:44.436122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:44.436097 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:44.442823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:44.442796 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:44.523444 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:44.523389 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podStartSLOduration=1.52337365 podStartE2EDuration="1.52337365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:44.521219696 +0000 UTC m=+3144.820629594" watchObservedRunningTime="2026-10-30 22:38:44.52337365 +0000 UTC m=+3144.822783524" Oct 30 22:38:45.415373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:45.415340 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:45.416509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:45.416482 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:38:47.911075 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:38:47.911041 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:38:53.168596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.168551 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz"] Oct 30 22:38:53.173379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.173354 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.226698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.226664 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz"] Oct 30 22:38:53.241738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.241706 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-74bb81\"/\"kube-root-ca.crt\"" Oct 30 22:38:53.241905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.241746 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-74bb81\"/\"openshift-service-ca.crt\"" Oct 30 22:38:53.268596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.268567 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-74bb81\"/\"default-dockercfg-rx6p4\"" Oct 30 22:38:53.268762 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.268641 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-74bb81\"/\"auth-test-model-74bb81-kserve-self-signed-certs\"" Oct 30 22:38:53.277195 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.277169 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7b68\" (UniqueName: \"kubernetes.io/projected/0d469a64-1f4f-43a0-b522-fe682affd4ba-kube-api-access-h7b68\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.277389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.277200 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d469a64-1f4f-43a0-b522-fe682affd4ba-tls-certs\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.277389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.277223 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-model-cache\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.277519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.277454 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-home\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.277519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.277475 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-dshm\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.277626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.277523 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-tmp-dir\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.381100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.381057 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7b68\" (UniqueName: \"kubernetes.io/projected/0d469a64-1f4f-43a0-b522-fe682affd4ba-kube-api-access-h7b68\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.382803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.382757 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d469a64-1f4f-43a0-b522-fe682affd4ba-tls-certs\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.382960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.382813 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-model-cache\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.383046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.383024 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-home\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.383223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.383203 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-dshm\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.383356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.383277 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-tmp-dir\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.384224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.383916 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-tmp-dir\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.384847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.384638 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-model-cache\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.385018 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.384985 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-home\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.399817 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.399784 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-dshm\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.400589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.400560 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d469a64-1f4f-43a0-b522-fe682affd4ba-tls-certs\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.440115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.440044 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7b68\" (UniqueName: \"kubernetes.io/projected/0d469a64-1f4f-43a0-b522-fe682affd4ba-kube-api-access-h7b68\") pod \"auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.486661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.486639 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:53.876194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:53.876161 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz"] Oct 30 22:38:53.876514 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:38:53.876490 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d469a64_1f4f_43a0_b522_fe682affd4ba.slice/crio-df641a1c343c7190a3709c917e082bfeecd60853930be2774cefc6e472ae6f27 WatchSource:0}: Error finding container df641a1c343c7190a3709c917e082bfeecd60853930be2774cefc6e472ae6f27: Status 404 returned error can't find the container with id df641a1c343c7190a3709c917e082bfeecd60853930be2774cefc6e472ae6f27 Oct 30 22:38:54.481655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:54.481620 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" event={"ID":"0d469a64-1f4f-43a0-b522-fe682affd4ba","Type":"ContainerStarted","Data":"b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e"} Oct 30 22:38:54.482045 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:54.481662 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" event={"ID":"0d469a64-1f4f-43a0-b522-fe682affd4ba","Type":"ContainerStarted","Data":"df641a1c343c7190a3709c917e082bfeecd60853930be2774cefc6e472ae6f27"} Oct 30 22:38:54.486855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:54.486842 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:38:54.597799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:38:54.597745 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" podStartSLOduration=1.5977303040000002 podStartE2EDuration="1.597730304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:54.595408674 +0000 UTC m=+3154.894818553" watchObservedRunningTime="2026-10-30 22:38:54.597730304 +0000 UTC m=+3154.897140183" Oct 30 22:39:03.487487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:03.487451 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:39:03.500198 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:03.500169 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:39:03.554228 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:03.554197 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:39:17.913657 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:39:17.913620 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:39:47.911143 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:39:47.911110 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:39:52.280733 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.280698 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm"] Oct 30 22:39:52.288623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.288600 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.338171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.338143 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-cbe3a9-serving-cert\"" Oct 30 22:39:52.390136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.390104 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm"] Oct 30 22:39:52.420360 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.417489 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nll6\" (UniqueName: \"kubernetes.io/projected/d972c878-20dd-426d-add5-ac94c39d35d5-kube-api-access-4nll6\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.420360 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.417643 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-metrics-certs\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.420360 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.417725 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-maas-api-tls\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.420360 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.417856 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d972c878-20dd-426d-add5-ac94c39d35d5-openshift-service-ca\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.518620 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.518589 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-maas-api-tls\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.518795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.518692 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d972c878-20dd-426d-add5-ac94c39d35d5-openshift-service-ca\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.518795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.518733 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nll6\" (UniqueName: \"kubernetes.io/projected/d972c878-20dd-426d-add5-ac94c39d35d5-kube-api-access-4nll6\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.518887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.518808 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-metrics-certs\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.519364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.519337 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d972c878-20dd-426d-add5-ac94c39d35d5-openshift-service-ca\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.521050 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.521026 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-maas-api-tls\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.521135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.521114 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-metrics-certs\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.576889 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.576827 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nll6\" (UniqueName: \"kubernetes.io/projected/d972c878-20dd-426d-add5-ac94c39d35d5-kube-api-access-4nll6\") pod \"maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:52.605778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:52.605754 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:53.019601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:53.019568 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm"] Oct 30 22:39:53.026129 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:39:53.026098 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd972c878_20dd_426d_add5_ac94c39d35d5.slice/crio-975eb3f1a10bf4ba7403394ba779c0fe77ba0850848275c60c490d5ac54b25e4 WatchSource:0}: Error finding container 975eb3f1a10bf4ba7403394ba779c0fe77ba0850848275c60c490d5ac54b25e4: Status 404 returned error can't find the container with id 975eb3f1a10bf4ba7403394ba779c0fe77ba0850848275c60c490d5ac54b25e4 Oct 30 22:39:53.027476 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:53.027457 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:39:53.908296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:53.908259 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" event={"ID":"d972c878-20dd-426d-add5-ac94c39d35d5","Type":"ContainerStarted","Data":"975eb3f1a10bf4ba7403394ba779c0fe77ba0850848275c60c490d5ac54b25e4"} Oct 30 22:39:54.917251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:54.917207 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" event={"ID":"d972c878-20dd-426d-add5-ac94c39d35d5","Type":"ContainerStarted","Data":"c2edf20ae6f3476a117d611311e2bee3ad7599e62669d7149c79849ae645de48"} Oct 30 22:39:54.918112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:54.918086 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:39:55.033559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:39:55.033490 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" podStartSLOduration=2.164110119 podStartE2EDuration="3.033471502s" totalImagesPullingTime="869.361383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:55.028916541 +0000 UTC m=+3215.328326423" watchObservedRunningTime="2026-10-30 22:39:55.033471502 +0000 UTC m=+3215.332881383" Oct 30 22:40:01.935641 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:01.935611 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:40:12.723706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:12.723669 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz"] Oct 30 22:40:17.913109 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:17.913055 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:40:25.200857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:25.200818 4991 generic.go:356] "Generic (PLEG): container finished" podID="67663820-6166-4317-a27d-83bd8d2e7843" containerID="aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7" exitCode=0 Oct 30 22:40:25.201422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:25.200898 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" event={"ID":"67663820-6166-4317-a27d-83bd8d2e7843","Type":"ContainerDied","Data":"aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7"} Oct 30 22:40:25.201422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:25.201377 4991 scope.go:122] "RemoveContainer" containerID="aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7" Oct 30 22:40:25.202900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:25.202867 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:40:25.203009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:25.202955 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:40:25.203075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:25.203005 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:40:26.211024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:26.210991 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" event={"ID":"67663820-6166-4317-a27d-83bd8d2e7843","Type":"ContainerStarted","Data":"bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae"} Oct 30 22:40:26.442689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:26.442648 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:40:26.448468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:26.448435 4991 patch_prober.go:29] interesting pod/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 22:40:26.449839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:26.449809 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 22:40:27.441351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:27.440676 4991 patch_prober.go:29] interesting pod/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 22:40:27.441351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:27.440739 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 22:40:27.753381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:27.750560 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" podUID="c96db307-5896-4311-93e0-4946a617469f" containerName="main" containerID="cri-o://33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903" gracePeriod=45 Oct 30 22:40:28.087952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.087927 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:40:28.228567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.218872 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-home\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-home\") pod \"c96db307-5896-4311-93e0-4946a617469f\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " Oct 30 22:40:28.228567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.219009 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96db307-5896-4311-93e0-4946a617469f-kube-api-access-t49f2\" (UniqueName: \"kubernetes.io/projected/c96db307-5896-4311-93e0-4946a617469f-kube-api-access-t49f2\") pod \"c96db307-5896-4311-93e0-4946a617469f\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " Oct 30 22:40:28.228567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.219106 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-dshm\") pod \"c96db307-5896-4311-93e0-4946a617469f\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " Oct 30 22:40:28.228567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.219143 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-tmp-dir\") pod \"c96db307-5896-4311-93e0-4946a617469f\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " Oct 30 22:40:28.228567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.219222 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c96db307-5896-4311-93e0-4946a617469f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c96db307-5896-4311-93e0-4946a617469f-tls-certs\") pod \"c96db307-5896-4311-93e0-4946a617469f\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " Oct 30 22:40:28.228567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.219290 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-model-cache\") pod \"c96db307-5896-4311-93e0-4946a617469f\" (UID: \"c96db307-5896-4311-93e0-4946a617469f\") " Oct 30 22:40:28.240349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.234624 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-home" pod "c96db307-5896-4311-93e0-4946a617469f" (UID: "c96db307-5896-4311-93e0-4946a617469f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:28.258484 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.258439 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-model-cache" pod "c96db307-5896-4311-93e0-4946a617469f" (UID: "c96db307-5896-4311-93e0-4946a617469f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:28.258664 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.258484 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-tmp-dir" pod "c96db307-5896-4311-93e0-4946a617469f" (UID: "c96db307-5896-4311-93e0-4946a617469f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:28.261026 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.260998 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c96db307-5896-4311-93e0-4946a617469f-tls-certs" pod "c96db307-5896-4311-93e0-4946a617469f" (UID: "c96db307-5896-4311-93e0-4946a617469f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:28.281259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.272709 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96db307-5896-4311-93e0-4946a617469f-kube-api-access-t49f2" pod "c96db307-5896-4311-93e0-4946a617469f" (UID: "c96db307-5896-4311-93e0-4946a617469f"). InnerVolumeSpecName "kube-api-access-t49f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:28.287862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.287840 4991 generic.go:356] "Generic (PLEG): container finished" podID="c96db307-5896-4311-93e0-4946a617469f" containerID="33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903" exitCode=0 Oct 30 22:40:28.288036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.288008 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" event={"ID":"c96db307-5896-4311-93e0-4946a617469f","Type":"ContainerDied","Data":"33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903"} Oct 30 22:40:28.288081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.288061 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" event={"ID":"c96db307-5896-4311-93e0-4946a617469f","Type":"ContainerDied","Data":"0922922ab561a84799673a5d74167e06ab77fbe7dd726b95849a9aaedc221812"} Oct 30 22:40:28.288117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.288084 4991 scope.go:122] "RemoveContainer" containerID="33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903" Oct 30 22:40:28.288565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.287920 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz" Oct 30 22:40:28.291624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.291597 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-dshm" pod "c96db307-5896-4311-93e0-4946a617469f" (UID: "c96db307-5896-4311-93e0-4946a617469f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:28.301384 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.301368 4991 scope.go:122] "RemoveContainer" containerID="33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903" Oct 30 22:40:28.301719 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:28.301697 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903\": container with ID starting with 33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903 not found: ID does not exist" containerID="33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903" Oct 30 22:40:28.301790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.301727 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903"} err="failed to get container status \"33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903\": rpc error: code = NotFound desc = could not find container \"33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903\": container with ID starting with 33f9f8d1b5e6f9f2a4cfd0cb8861df756002c4fa47f0ffe01d0d0cf125bf5903 not found: ID does not exist" Oct 30 22:40:28.351392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.344363 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t49f2\" (UniqueName: \"kubernetes.io/projected/c96db307-5896-4311-93e0-4946a617469f-kube-api-access-t49f2\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.351601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.351581 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.351708 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.351695 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.351802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.351790 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96db307-5896-4311-93e0-4946a617469f-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.351893 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.351877 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.351988 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.351975 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c96db307-5896-4311-93e0-4946a617469f-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.436644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.436614 4991 patch_prober.go:29] interesting pod/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 22:40:28.436853 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.436827 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 22:40:28.707289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.707209 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz"] Oct 30 22:40:28.712097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:28.712072 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9717a8/auth-test-model-9717a8-kserve-678787c4f7-crjcz"] Oct 30 22:40:29.438344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:29.435648 4991 patch_prober.go:29] interesting pod/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 22:40:29.438344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:29.435724 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 22:40:29.879376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:29.879341 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96db307-5896-4311-93e0-4946a617469f" path="/var/lib/kubelet/pods/c96db307-5896-4311-93e0-4946a617469f/volumes" Oct 30 22:40:30.415397 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:30.415373 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:40:30.438702 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:30.438665 4991 patch_prober.go:29] interesting pod/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 22:40:30.438975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:30.438937 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 22:40:31.436283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:31.436243 4991 patch_prober.go:29] interesting pod/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 22:40:31.436704 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:31.436312 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 22:40:32.440536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:32.440504 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:40:32.441364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:32.441341 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:40:43.550871 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.550835 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-2ecef3"] Oct 30 22:40:43.551709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.551688 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c96db307-5896-4311-93e0-4946a617469f" containerName="main" Oct 30 22:40:43.551768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.551712 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96db307-5896-4311-93e0-4946a617469f" containerName="main" Oct 30 22:40:43.551889 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.551878 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96db307-5896-4311-93e0-4946a617469f" containerName="main" Oct 30 22:40:43.557374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.557347 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:43.606553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.606530 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-2ecef3"] Oct 30 22:40:43.648820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.648779 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rll4p\" (UniqueName: \"kubernetes.io/projected/c2c67ce6-ba26-4eab-9337-8a848a517b7c-kube-api-access-rll4p\") pod \"test-curl-7706-2ecef3\" (UID: \"c2c67ce6-ba26-4eab-9337-8a848a517b7c\") " pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:43.648991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.648856 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmbvp\"" Oct 30 22:40:43.750076 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.750038 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rll4p\" (UniqueName: \"kubernetes.io/projected/c2c67ce6-ba26-4eab-9337-8a848a517b7c-kube-api-access-rll4p\") pod \"test-curl-7706-2ecef3\" (UID: \"c2c67ce6-ba26-4eab-9337-8a848a517b7c\") " pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:43.808371 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.808283 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rll4p\" (UniqueName: \"kubernetes.io/projected/c2c67ce6-ba26-4eab-9337-8a848a517b7c-kube-api-access-rll4p\") pod \"test-curl-7706-2ecef3\" (UID: \"c2c67ce6-ba26-4eab-9337-8a848a517b7c\") " pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:43.869959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:43.869933 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:44.247755 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:44.247718 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-2ecef3"] Oct 30 22:40:44.252976 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:40:44.252940 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2c67ce6_ba26_4eab_9337_8a848a517b7c.slice/crio-2ff0ad38ef61b92b595a3774277d59668d2ff3ecfa743ce0abb03b8d4ba040ea WatchSource:0}: Error finding container 2ff0ad38ef61b92b595a3774277d59668d2ff3ecfa743ce0abb03b8d4ba040ea: Status 404 returned error can't find the container with id 2ff0ad38ef61b92b595a3774277d59668d2ff3ecfa743ce0abb03b8d4ba040ea Oct 30 22:40:44.406280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:44.406248 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-2ecef3" event={"ID":"c2c67ce6-ba26-4eab-9337-8a848a517b7c","Type":"ContainerStarted","Data":"52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf"} Oct 30 22:40:44.406426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:44.406294 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-2ecef3" event={"ID":"c2c67ce6-ba26-4eab-9337-8a848a517b7c","Type":"ContainerStarted","Data":"2ff0ad38ef61b92b595a3774277d59668d2ff3ecfa743ce0abb03b8d4ba040ea"} Oct 30 22:40:44.518986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:44.518902 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7706-2ecef3" podStartSLOduration=1.518890029 podStartE2EDuration="1.518890029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:44.516572344 +0000 UTC m=+3264.815982227" watchObservedRunningTime="2026-10-30 22:40:44.518890029 +0000 UTC m=+3264.818299909" Oct 30 22:40:46.019159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.019123 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-2ecef3"] Oct 30 22:40:46.024156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.024124 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-2ecef3"] Oct 30 22:40:46.183489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.183458 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-7865fd"] Oct 30 22:40:46.184017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.184002 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" containerName="test-curl-7706-2ecef3" Oct 30 22:40:46.184068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.184020 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" containerName="test-curl-7706-2ecef3" Oct 30 22:40:46.184123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.184114 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" containerName="test-curl-7706-2ecef3" Oct 30 22:40:46.187381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.187365 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:46.241286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.241256 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-7865fd"] Oct 30 22:40:46.278840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.278779 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blhf\" (UniqueName: \"kubernetes.io/projected/bddec78b-7299-41f6-a759-febe7f2f203f-kube-api-access-6blhf\") pod \"test-curl-7706-7865fd\" (UID: \"bddec78b-7299-41f6-a759-febe7f2f203f\") " pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:46.380256 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.380227 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blhf\" (UniqueName: \"kubernetes.io/projected/bddec78b-7299-41f6-a759-febe7f2f203f-kube-api-access-6blhf\") pod \"test-curl-7706-7865fd\" (UID: \"bddec78b-7299-41f6-a759-febe7f2f203f\") " pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:46.418491 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.418436 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7706-2ecef3" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" containerName="test-curl-7706-2ecef3" containerID="cri-o://52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf" gracePeriod=2 Oct 30 22:40:46.437956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.437925 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blhf\" (UniqueName: \"kubernetes.io/projected/bddec78b-7299-41f6-a759-febe7f2f203f-kube-api-access-6blhf\") pod \"test-curl-7706-7865fd\" (UID: \"bddec78b-7299-41f6-a759-febe7f2f203f\") " pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:46.466327 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:46.466283 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-2ecef3\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:46.510428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.510402 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:46.896123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:46.896051 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-7865fd"] Oct 30 22:40:46.902241 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:40:46.902211 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbddec78b_7299_41f6_a759_febe7f2f203f.slice/crio-fc71f46e6e0d5de82430c7ba1ff4d155f850806ac7127424e423adbd450b36fa WatchSource:0}: Error finding container fc71f46e6e0d5de82430c7ba1ff4d155f850806ac7127424e423adbd450b36fa: Status 404 returned error can't find the container with id fc71f46e6e0d5de82430c7ba1ff4d155f850806ac7127424e423adbd450b36fa Oct 30 22:40:47.427085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:47.427032 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-7865fd" event={"ID":"bddec78b-7299-41f6-a759-febe7f2f203f","Type":"ContainerStarted","Data":"293ac0b9827485ff75fdf369d30c60fa57c735657cfa4b42c08fc94962e557b5"} Oct 30 22:40:47.427500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:47.427097 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-7865fd" event={"ID":"bddec78b-7299-41f6-a759-febe7f2f203f","Type":"ContainerStarted","Data":"fc71f46e6e0d5de82430c7ba1ff4d155f850806ac7127424e423adbd450b36fa"} Oct 30 22:40:47.475018 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:47.474986 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-2ecef3\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:47.585496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:47.585444 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7706-7865fd" podStartSLOduration=1.5854272790000001 podStartE2EDuration="1.585427279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:47.582243973 +0000 UTC m=+3267.881653853" watchObservedRunningTime="2026-10-30 22:40:47.585427279 +0000 UTC m=+3267.884837160" Oct 30 22:40:47.911489 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:47.911448 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:40:48.693518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:48.693460 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:48.805139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:48.805104 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c67ce6-ba26-4eab-9337-8a848a517b7c-kube-api-access-rll4p\" (UniqueName: \"kubernetes.io/projected/c2c67ce6-ba26-4eab-9337-8a848a517b7c-kube-api-access-rll4p\") pod \"c2c67ce6-ba26-4eab-9337-8a848a517b7c\" (UID: \"c2c67ce6-ba26-4eab-9337-8a848a517b7c\") " Oct 30 22:40:48.807188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:48.807154 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c67ce6-ba26-4eab-9337-8a848a517b7c-kube-api-access-rll4p" pod "c2c67ce6-ba26-4eab-9337-8a848a517b7c" (UID: "c2c67ce6-ba26-4eab-9337-8a848a517b7c"). InnerVolumeSpecName "kube-api-access-rll4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:48.906846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:48.906820 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rll4p\" (UniqueName: \"kubernetes.io/projected/c2c67ce6-ba26-4eab-9337-8a848a517b7c-kube-api-access-rll4p\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:49.326184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.326148 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-7865fd"] Oct 30 22:40:49.331432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.331391 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-7865fd"] Oct 30 22:40:49.442861 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.442824 4991 generic.go:356] "Generic (PLEG): container finished" podID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" containerID="52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf" exitCode=137 Oct 30 22:40:49.443006 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.442888 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:49.443006 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.442924 4991 scope.go:122] "RemoveContainer" containerID="52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf" Oct 30 22:40:49.443153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.443130 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7706-7865fd" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" containerName="test-curl-7706-7865fd" containerID="cri-o://293ac0b9827485ff75fdf369d30c60fa57c735657cfa4b42c08fc94962e557b5" gracePeriod=2 Oct 30 22:40:49.454766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.454749 4991 scope.go:122] "RemoveContainer" containerID="52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf" Oct 30 22:40:49.455082 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:49.455053 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf\": container with ID starting with 52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf not found: ID does not exist" containerID="52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf" Oct 30 22:40:49.455203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.455090 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf"} err="failed to get container status \"52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf\": rpc error: code = NotFound desc = could not find container \"52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf\": container with ID starting with 52529648c73e2e27da2a4ad27bdaf871a633fe222238a3724def87cf07c4b4bf not found: ID does not exist" Oct 30 22:40:49.491240 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:49.491204 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:49.539189 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:49.539155 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-2ecef3\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:49.587100 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:49.587032 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:49.710386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.710358 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-fa1a75"] Oct 30 22:40:49.710973 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.710957 4991 cpu_manager.go:414] "containerMap: removing container" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" containerName="test-curl-7706-7865fd" Oct 30 22:40:49.711024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.710975 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" containerName="test-curl-7706-7865fd" Oct 30 22:40:49.711108 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.711099 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" containerName="test-curl-7706-7865fd" Oct 30 22:40:49.715800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.715779 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:49.769586 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.769560 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-fa1a75"] Oct 30 22:40:49.815919 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.815892 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjg5\" (UniqueName: \"kubernetes.io/projected/25012d35-96dd-4ed6-a072-292f87b00615-kube-api-access-9zjg5\") pod \"test-curl-7706-fa1a75\" (UID: \"25012d35-96dd-4ed6-a072-292f87b00615\") " pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:49.872932 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:49.872857 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-2ecef3\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" pod="opendatahub/test-curl-7706-2ecef3" Oct 30 22:40:49.877047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.877023 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c67ce6-ba26-4eab-9337-8a848a517b7c" path="/var/lib/kubelet/pods/c2c67ce6-ba26-4eab-9337-8a848a517b7c/volumes" Oct 30 22:40:49.917229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.917204 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjg5\" (UniqueName: \"kubernetes.io/projected/25012d35-96dd-4ed6-a072-292f87b00615-kube-api-access-9zjg5\") pod \"test-curl-7706-fa1a75\" (UID: \"25012d35-96dd-4ed6-a072-292f87b00615\") " pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:49.920626 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:49.920597 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:49.975615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:49.975590 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjg5\" (UniqueName: \"kubernetes.io/projected/25012d35-96dd-4ed6-a072-292f87b00615-kube-api-access-9zjg5\") pod \"test-curl-7706-fa1a75\" (UID: \"25012d35-96dd-4ed6-a072-292f87b00615\") " pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:50.028311 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:50.028277 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:50.406462 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:50.406422 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-fa1a75"] Oct 30 22:40:50.412506 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:40:50.412477 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25012d35_96dd_4ed6_a072_292f87b00615.slice/crio-45aa002927d99ca981628aec8300e407c6e961616f03f150464493b227a046b6 WatchSource:0}: Error finding container 45aa002927d99ca981628aec8300e407c6e961616f03f150464493b227a046b6: Status 404 returned error can't find the container with id 45aa002927d99ca981628aec8300e407c6e961616f03f150464493b227a046b6 Oct 30 22:40:50.456956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:50.456928 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-fa1a75" event={"ID":"25012d35-96dd-4ed6-a072-292f87b00615","Type":"ContainerStarted","Data":"45aa002927d99ca981628aec8300e407c6e961616f03f150464493b227a046b6"} Oct 30 22:40:51.467940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.467911 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-fa1a75" event={"ID":"25012d35-96dd-4ed6-a072-292f87b00615","Type":"ContainerStarted","Data":"512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611"} Oct 30 22:40:51.469599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.469575 4991 generic.go:356] "Generic (PLEG): container finished" podID="bddec78b-7299-41f6-a759-febe7f2f203f" containerID="293ac0b9827485ff75fdf369d30c60fa57c735657cfa4b42c08fc94962e557b5" exitCode=137 Oct 30 22:40:51.583684 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.583608 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7706-fa1a75" podStartSLOduration=2.583593975 podStartE2EDuration="2.583593975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:51.57880257 +0000 UTC m=+3271.878212475" watchObservedRunningTime="2026-10-30 22:40:51.583593975 +0000 UTC m=+3271.883003854" Oct 30 22:40:51.679952 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.679929 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:51.727807 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:51.727769 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:51.748276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.748247 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddec78b-7299-41f6-a759-febe7f2f203f-kube-api-access-6blhf\" (UniqueName: \"kubernetes.io/projected/bddec78b-7299-41f6-a759-febe7f2f203f-kube-api-access-6blhf\") pod \"bddec78b-7299-41f6-a759-febe7f2f203f\" (UID: \"bddec78b-7299-41f6-a759-febe7f2f203f\") " Oct 30 22:40:51.750470 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.750428 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddec78b-7299-41f6-a759-febe7f2f203f-kube-api-access-6blhf" pod "bddec78b-7299-41f6-a759-febe7f2f203f" (UID: "bddec78b-7299-41f6-a759-febe7f2f203f"). InnerVolumeSpecName "kube-api-access-6blhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:51.850089 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.850050 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blhf\" (UniqueName: \"kubernetes.io/projected/bddec78b-7299-41f6-a759-febe7f2f203f-kube-api-access-6blhf\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:51.879568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:51.879531 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" path="/var/lib/kubelet/pods/bddec78b-7299-41f6-a759-febe7f2f203f/volumes" Oct 30 22:40:52.487569 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:52.487519 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:52.488076 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:52.487854 4991 scope.go:122] "RemoveContainer" containerID="293ac0b9827485ff75fdf369d30c60fa57c735657cfa4b42c08fc94962e557b5" Oct 30 22:40:52.536132 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:52.535898 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:52.592361 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:52.589760 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:52.995949 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:52.989084 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-fa1a75"] Oct 30 22:40:53.037862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.036111 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-fa1a75"] Oct 30 22:40:53.175382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.175341 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-a386f9"] Oct 30 22:40:53.176286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.176256 4991 cpu_manager.go:414] "containerMap: removing container" podUID="25012d35-96dd-4ed6-a072-292f87b00615" containerName="test-curl-7706-fa1a75" Oct 30 22:40:53.176286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.176283 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25012d35-96dd-4ed6-a072-292f87b00615" containerName="test-curl-7706-fa1a75" Oct 30 22:40:53.176482 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.176464 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="25012d35-96dd-4ed6-a072-292f87b00615" containerName="test-curl-7706-fa1a75" Oct 30 22:40:53.181897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.181875 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:53.234690 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:53.232833 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:53.249287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.248998 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-a386f9"] Oct 30 22:40:53.281280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.281240 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllnd\" (UniqueName: \"kubernetes.io/projected/99d5bd0f-c6ba-4a1c-8c2d-773119585d41-kube-api-access-qllnd\") pod \"test-curl-7706-a386f9\" (UID: \"99d5bd0f-c6ba-4a1c-8c2d-773119585d41\") " pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:53.382712 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.382675 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllnd\" (UniqueName: \"kubernetes.io/projected/99d5bd0f-c6ba-4a1c-8c2d-773119585d41-kube-api-access-qllnd\") pod \"test-curl-7706-a386f9\" (UID: \"99d5bd0f-c6ba-4a1c-8c2d-773119585d41\") " pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:53.440719 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.440680 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllnd\" (UniqueName: \"kubernetes.io/projected/99d5bd0f-c6ba-4a1c-8c2d-773119585d41-kube-api-access-qllnd\") pod \"test-curl-7706-a386f9\" (UID: \"99d5bd0f-c6ba-4a1c-8c2d-773119585d41\") " pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:53.494619 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.494577 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7706-fa1a75" podUID="25012d35-96dd-4ed6-a072-292f87b00615" containerName="test-curl-7706-fa1a75" containerID="cri-o://512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611" gracePeriod=2 Oct 30 22:40:53.495030 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.494674 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:53.542689 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:53.542655 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7865fd\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="bddec78b-7299-41f6-a759-febe7f2f203f" pod="opendatahub/test-curl-7706-7865fd" Oct 30 22:40:53.590744 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:53.590565 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-fa1a75\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="25012d35-96dd-4ed6-a072-292f87b00615" pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:53.889299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:53.889216 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-a386f9"] Oct 30 22:40:53.894397 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:40:53.894361 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99d5bd0f_c6ba_4a1c_8c2d_773119585d41.slice/crio-d0cf37389488bca8ff1c5d14e5a464fea1b749739f3c045e1032c82425768061 WatchSource:0}: Error finding container d0cf37389488bca8ff1c5d14e5a464fea1b749739f3c045e1032c82425768061: Status 404 returned error can't find the container with id d0cf37389488bca8ff1c5d14e5a464fea1b749739f3c045e1032c82425768061 Oct 30 22:40:54.504434 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:54.504394 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-a386f9" event={"ID":"99d5bd0f-c6ba-4a1c-8c2d-773119585d41","Type":"ContainerStarted","Data":"718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920"} Oct 30 22:40:54.504926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:54.504445 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-a386f9" event={"ID":"99d5bd0f-c6ba-4a1c-8c2d-773119585d41","Type":"ContainerStarted","Data":"d0cf37389488bca8ff1c5d14e5a464fea1b749739f3c045e1032c82425768061"} Oct 30 22:40:54.614118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:54.614065 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7706-a386f9" podStartSLOduration=1.614051102 podStartE2EDuration="1.614051102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:54.611244747 +0000 UTC m=+3274.910654627" watchObservedRunningTime="2026-10-30 22:40:54.614051102 +0000 UTC m=+3274.913460981" Oct 30 22:40:55.767695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:55.767625 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:55.816374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:55.816341 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25012d35-96dd-4ed6-a072-292f87b00615-kube-api-access-9zjg5\" (UniqueName: \"kubernetes.io/projected/25012d35-96dd-4ed6-a072-292f87b00615-kube-api-access-9zjg5\") pod \"25012d35-96dd-4ed6-a072-292f87b00615\" (UID: \"25012d35-96dd-4ed6-a072-292f87b00615\") " Oct 30 22:40:55.818357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:55.818306 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25012d35-96dd-4ed6-a072-292f87b00615-kube-api-access-9zjg5" pod "25012d35-96dd-4ed6-a072-292f87b00615" (UID: "25012d35-96dd-4ed6-a072-292f87b00615"). InnerVolumeSpecName "kube-api-access-9zjg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:55.875682 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:55.875654 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25012d35-96dd-4ed6-a072-292f87b00615" path="/var/lib/kubelet/pods/25012d35-96dd-4ed6-a072-292f87b00615/volumes" Oct 30 22:40:55.917759 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:55.917725 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjg5\" (UniqueName: \"kubernetes.io/projected/25012d35-96dd-4ed6-a072-292f87b00615-kube-api-access-9zjg5\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.118633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.118541 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-a386f9"] Oct 30 22:40:56.125517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.125483 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-a386f9"] Oct 30 22:40:56.286018 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.285985 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-7e4494"] Oct 30 22:40:56.286694 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.286670 4991 cpu_manager.go:414] "containerMap: removing container" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" containerName="test-curl-7706-a386f9" Oct 30 22:40:56.286820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.286694 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" containerName="test-curl-7706-a386f9" Oct 30 22:40:56.286882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.286844 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" containerName="test-curl-7706-a386f9" Oct 30 22:40:56.290529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.290506 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:56.348935 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.348908 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-7e4494"] Oct 30 22:40:56.424760 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.424686 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqjb\" (UniqueName: \"kubernetes.io/projected/524e86ed-385c-424e-b258-bc315a48a99c-kube-api-access-cvqjb\") pod \"test-curl-7706-7e4494\" (UID: \"524e86ed-385c-424e-b258-bc315a48a99c\") " pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:56.521670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.521635 4991 generic.go:356] "Generic (PLEG): container finished" podID="25012d35-96dd-4ed6-a072-292f87b00615" containerID="512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611" exitCode=137 Oct 30 22:40:56.521828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.521680 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fa1a75" Oct 30 22:40:56.521828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.521731 4991 scope.go:122] "RemoveContainer" containerID="512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611" Oct 30 22:40:56.521932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.521887 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7706-a386f9" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" containerName="test-curl-7706-a386f9" containerID="cri-o://718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920" gracePeriod=2 Oct 30 22:40:56.525956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.525929 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqjb\" (UniqueName: \"kubernetes.io/projected/524e86ed-385c-424e-b258-bc315a48a99c-kube-api-access-cvqjb\") pod \"test-curl-7706-7e4494\" (UID: \"524e86ed-385c-424e-b258-bc315a48a99c\") " pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:56.531538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.531515 4991 scope.go:122] "RemoveContainer" containerID="512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611" Oct 30 22:40:56.531787 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:56.531768 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611\": container with ID starting with 512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611 not found: ID does not exist" containerID="512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611" Oct 30 22:40:56.531870 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.531794 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611"} err="failed to get container status \"512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611\": rpc error: code = NotFound desc = could not find container \"512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611\": container with ID starting with 512c9311942b391e039dd294ce7d302b9f8a011cf3de3237caff05ffd7514611 not found: ID does not exist" Oct 30 22:40:56.569978 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:56.569939 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-a386f9\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:56.584957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.584933 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqjb\" (UniqueName: \"kubernetes.io/projected/524e86ed-385c-424e-b258-bc315a48a99c-kube-api-access-cvqjb\") pod \"test-curl-7706-7e4494\" (UID: \"524e86ed-385c-424e-b258-bc315a48a99c\") " pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:56.604912 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.604889 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:56.619992 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:56.619233 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-a386f9\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:56.990424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:56.990391 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-7e4494"] Oct 30 22:40:56.996446 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:40:56.996416 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod524e86ed_385c_424e_b258_bc315a48a99c.slice/crio-55f9aac3746b94b37c252d22a2a480b844928a41e961d50fdc676421ca345633 WatchSource:0}: Error finding container 55f9aac3746b94b37c252d22a2a480b844928a41e961d50fdc676421ca345633: Status 404 returned error can't find the container with id 55f9aac3746b94b37c252d22a2a480b844928a41e961d50fdc676421ca345633 Oct 30 22:40:57.531421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:57.531383 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-7e4494" event={"ID":"524e86ed-385c-424e-b258-bc315a48a99c","Type":"ContainerStarted","Data":"aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be"} Oct 30 22:40:57.531608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:57.531435 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-7e4494" event={"ID":"524e86ed-385c-424e-b258-bc315a48a99c","Type":"ContainerStarted","Data":"55f9aac3746b94b37c252d22a2a480b844928a41e961d50fdc676421ca345633"} Oct 30 22:40:57.645591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:57.645544 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7706-7e4494" podStartSLOduration=1.645532161 podStartE2EDuration="1.645532161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:57.645350181 +0000 UTC m=+3277.944760059" watchObservedRunningTime="2026-10-30 22:40:57.645532161 +0000 UTC m=+3277.944942067" Oct 30 22:40:57.693084 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:57.693049 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-a386f9\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:58.766131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:58.766105 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:58.853833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:58.853769 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d5bd0f-c6ba-4a1c-8c2d-773119585d41-kube-api-access-qllnd\" (UniqueName: \"kubernetes.io/projected/99d5bd0f-c6ba-4a1c-8c2d-773119585d41-kube-api-access-qllnd\") pod \"99d5bd0f-c6ba-4a1c-8c2d-773119585d41\" (UID: \"99d5bd0f-c6ba-4a1c-8c2d-773119585d41\") " Oct 30 22:40:58.855788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:58.855757 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d5bd0f-c6ba-4a1c-8c2d-773119585d41-kube-api-access-qllnd" pod "99d5bd0f-c6ba-4a1c-8c2d-773119585d41" (UID: "99d5bd0f-c6ba-4a1c-8c2d-773119585d41"). InnerVolumeSpecName "kube-api-access-qllnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:58.955828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:58.955804 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllnd\" (UniqueName: \"kubernetes.io/projected/99d5bd0f-c6ba-4a1c-8c2d-773119585d41-kube-api-access-qllnd\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:59.139730 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.139645 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-7e4494"] Oct 30 22:40:59.145410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.145374 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-7e4494"] Oct 30 22:40:59.308283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.308249 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-fc6448"] Oct 30 22:40:59.308856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.308836 4991 cpu_manager.go:414] "containerMap: removing container" podUID="524e86ed-385c-424e-b258-bc315a48a99c" containerName="test-curl-7706-7e4494" Oct 30 22:40:59.308901 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.308858 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524e86ed-385c-424e-b258-bc315a48a99c" containerName="test-curl-7706-7e4494" Oct 30 22:40:59.309000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.308990 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="524e86ed-385c-424e-b258-bc315a48a99c" containerName="test-curl-7706-7e4494" Oct 30 22:40:59.312442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.312422 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:40:59.363011 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.362988 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-fc6448"] Oct 30 22:40:59.461879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.461803 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2r6\" (UniqueName: \"kubernetes.io/projected/ab138fa6-0f56-41e8-a626-1dca0d59e18c-kube-api-access-7n2r6\") pod \"test-curl-7706-fc6448\" (UID: \"ab138fa6-0f56-41e8-a626-1dca0d59e18c\") " pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:40:59.547462 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.547435 4991 generic.go:356] "Generic (PLEG): container finished" podID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" containerID="718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920" exitCode=137 Oct 30 22:40:59.547608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.547484 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:59.547608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.547538 4991 scope.go:122] "RemoveContainer" containerID="718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920" Oct 30 22:40:59.547763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.547740 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7706-7e4494" podUID="524e86ed-385c-424e-b258-bc315a48a99c" containerName="test-curl-7706-7e4494" containerID="cri-o://aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be" gracePeriod=2 Oct 30 22:40:59.558171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.558156 4991 scope.go:122] "RemoveContainer" containerID="718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920" Oct 30 22:40:59.558471 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:59.558448 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920\": container with ID starting with 718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920 not found: ID does not exist" containerID="718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920" Oct 30 22:40:59.558535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.558483 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920"} err="failed to get container status \"718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920\": rpc error: code = NotFound desc = could not find container \"718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920\": container with ID starting with 718bb1d2cb044f650db4eba2bcbf629d3a397162ab4c6b811714317f22070920 not found: ID does not exist" Oct 30 22:40:59.562411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.562390 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2r6\" (UniqueName: \"kubernetes.io/projected/ab138fa6-0f56-41e8-a626-1dca0d59e18c-kube-api-access-7n2r6\") pod \"test-curl-7706-fc6448\" (UID: \"ab138fa6-0f56-41e8-a626-1dca0d59e18c\") " pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:40:59.600327 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:59.600294 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7e4494\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="524e86ed-385c-424e-b258-bc315a48a99c" pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:59.620221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.620198 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2r6\" (UniqueName: \"kubernetes.io/projected/ab138fa6-0f56-41e8-a626-1dca0d59e18c-kube-api-access-7n2r6\") pod \"test-curl-7706-fc6448\" (UID: \"ab138fa6-0f56-41e8-a626-1dca0d59e18c\") " pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:40:59.625793 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.625772 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:40:59.648568 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:59.648542 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7e4494\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="524e86ed-385c-424e-b258-bc315a48a99c" pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:59.696497 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:59.696455 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-a386f9\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:40:59.874941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:40:59.874875 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" path="/var/lib/kubelet/pods/99d5bd0f-c6ba-4a1c-8c2d-773119585d41/volumes" Oct 30 22:40:59.919027 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:59.918997 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-7e4494\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="524e86ed-385c-424e-b258-bc315a48a99c" pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:40:59.966707 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:40:59.966673 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-a386f9\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="99d5bd0f-c6ba-4a1c-8c2d-773119585d41" pod="opendatahub/test-curl-7706-a386f9" Oct 30 22:41:00.004821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:00.004794 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-fc6448"] Oct 30 22:41:00.010878 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:41:00.010843 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab138fa6_0f56_41e8_a626_1dca0d59e18c.slice/crio-cdc5b25365e1e75d27ac7fabfaad8d647741e4634209476b87f9ccc0a8ed3d79 WatchSource:0}: Error finding container cdc5b25365e1e75d27ac7fabfaad8d647741e4634209476b87f9ccc0a8ed3d79: Status 404 returned error can't find the container with id cdc5b25365e1e75d27ac7fabfaad8d647741e4634209476b87f9ccc0a8ed3d79 Oct 30 22:41:00.556532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:00.556483 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-fc6448" event={"ID":"ab138fa6-0f56-41e8-a626-1dca0d59e18c","Type":"ContainerStarted","Data":"f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397"} Oct 30 22:41:00.556778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:00.556543 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-fc6448" event={"ID":"ab138fa6-0f56-41e8-a626-1dca0d59e18c","Type":"ContainerStarted","Data":"cdc5b25365e1e75d27ac7fabfaad8d647741e4634209476b87f9ccc0a8ed3d79"} Oct 30 22:41:00.673123 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:00.673075 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7706-fc6448" podStartSLOduration=1.6730631649999999 podStartE2EDuration="1.673063165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:00.670467076 +0000 UTC m=+3280.969876938" watchObservedRunningTime="2026-10-30 22:41:00.673063165 +0000 UTC m=+3280.972473045" Oct 30 22:41:01.809658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:01.809562 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:41:01.887250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:01.887218 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/524e86ed-385c-424e-b258-bc315a48a99c-kube-api-access-cvqjb\" (UniqueName: \"kubernetes.io/projected/524e86ed-385c-424e-b258-bc315a48a99c-kube-api-access-cvqjb\") pod \"524e86ed-385c-424e-b258-bc315a48a99c\" (UID: \"524e86ed-385c-424e-b258-bc315a48a99c\") " Oct 30 22:41:01.889362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:01.889312 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524e86ed-385c-424e-b258-bc315a48a99c-kube-api-access-cvqjb" pod "524e86ed-385c-424e-b258-bc315a48a99c" (UID: "524e86ed-385c-424e-b258-bc315a48a99c"). InnerVolumeSpecName "kube-api-access-cvqjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:01.989125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:01.989090 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqjb\" (UniqueName: \"kubernetes.io/projected/524e86ed-385c-424e-b258-bc315a48a99c-kube-api-access-cvqjb\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.183095 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.183003 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-fc6448"] Oct 30 22:41:02.189404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.189371 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-fc6448"] Oct 30 22:41:02.586764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.586711 4991 generic.go:356] "Generic (PLEG): container finished" podID="524e86ed-385c-424e-b258-bc315a48a99c" containerID="aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be" exitCode=137 Oct 30 22:41:02.586764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.586752 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-7e4494" Oct 30 22:41:02.587014 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.586820 4991 scope.go:122] "RemoveContainer" containerID="aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be" Oct 30 22:41:02.587014 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.586893 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7706-fc6448" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" containerName="test-curl-7706-fc6448" containerID="cri-o://f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397" gracePeriod=2 Oct 30 22:41:02.597366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.597344 4991 scope.go:122] "RemoveContainer" containerID="aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be" Oct 30 22:41:02.597626 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:02.597607 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be\": container with ID starting with aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be not found: ID does not exist" containerID="aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be" Oct 30 22:41:02.597677 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:02.597634 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be"} err="failed to get container status \"aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be\": rpc error: code = NotFound desc = could not find container \"aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be\": container with ID starting with aa39ea4a8da99768fb5927dc35c93a7ff8d25bab29b54c2650035d446cc0d0be not found: ID does not exist" Oct 30 22:41:02.634992 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:02.634954 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-fc6448\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:41:02.682747 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:02.682709 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-fc6448\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:41:03.875826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:03.875796 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="524e86ed-385c-424e-b258-bc315a48a99c" path="/var/lib/kubelet/pods/524e86ed-385c-424e-b258-bc315a48a99c/volumes" Oct 30 22:41:04.799425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.799396 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:41:04.820601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.820576 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw"] Oct 30 22:41:04.821250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.821232 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" containerName="test-curl-7706-fc6448" Oct 30 22:41:04.821300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.821252 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" containerName="test-curl-7706-fc6448" Oct 30 22:41:04.821475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.821461 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" containerName="test-curl-7706-fc6448" Oct 30 22:41:04.824809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.824793 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.882668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.882594 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-421286-openshift-default-dockercfg-qlszb\"" Oct 30 22:41:04.931591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.931563 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw"] Oct 30 22:41:04.931826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.931750 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab138fa6-0f56-41e8-a626-1dca0d59e18c-kube-api-access-7n2r6\" (UniqueName: \"kubernetes.io/projected/ab138fa6-0f56-41e8-a626-1dca0d59e18c-kube-api-access-7n2r6\") pod \"ab138fa6-0f56-41e8-a626-1dca0d59e18c\" (UID: \"ab138fa6-0f56-41e8-a626-1dca0d59e18c\") " Oct 30 22:41:04.932208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932184 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.932369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932229 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.932369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932304 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d126b-8220-489e-91df-595b1bdce9b6-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.932369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932357 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.932536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932399 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.932536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932478 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.932536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932519 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5fdj\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-kube-api-access-s5fdj\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.932714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932686 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.933337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.932737 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d126b-8220-489e-91df-595b1bdce9b6-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:04.954559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:04.952041 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab138fa6-0f56-41e8-a626-1dca0d59e18c-kube-api-access-7n2r6" pod "ab138fa6-0f56-41e8-a626-1dca0d59e18c" (UID: "ab138fa6-0f56-41e8-a626-1dca0d59e18c"). InnerVolumeSpecName "kube-api-access-7n2r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:05.034220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034176 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034221 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5fdj\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-kube-api-access-s5fdj\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034358 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034409 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d126b-8220-489e-91df-595b1bdce9b6-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034476 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034617 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034705 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d126b-8220-489e-91df-595b1bdce9b6-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034739 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.034859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034796 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.035017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.034864 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.035080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.035018 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.035304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.035277 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.035696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.035667 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2r6\" (UniqueName: \"kubernetes.io/projected/ab138fa6-0f56-41e8-a626-1dca0d59e18c-kube-api-access-7n2r6\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:05.035812 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.035668 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.035882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.035838 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d126b-8220-489e-91df-595b1bdce9b6-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.037990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.037961 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.038166 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.038145 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d126b-8220-489e-91df-595b1bdce9b6-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.097715 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.097687 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5fdj\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-kube-api-access-s5fdj\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.098010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.097987 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.143661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.143590 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:05.610486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.610452 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw"] Oct 30 22:41:05.613561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.613529 4991 generic.go:356] "Generic (PLEG): container finished" podID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" containerID="f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397" exitCode=137 Oct 30 22:41:05.613698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.613573 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:41:05.613698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.613622 4991 scope.go:122] "RemoveContainer" containerID="f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397" Oct 30 22:41:05.617442 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:41:05.617415 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768d126b_8220_489e_91df_595b1bdce9b6.slice/crio-ba95dd2a2f99d06c484daaab5c2564735388b4fd85b204f34baa107ad4e89067 WatchSource:0}: Error finding container ba95dd2a2f99d06c484daaab5c2564735388b4fd85b204f34baa107ad4e89067: Status 404 returned error can't find the container with id ba95dd2a2f99d06c484daaab5c2564735388b4fd85b204f34baa107ad4e89067 Oct 30 22:41:05.619593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.619559 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:41:05.619682 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.619642 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:41:05.619731 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.619686 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:41:05.634655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.634633 4991 scope.go:122] "RemoveContainer" containerID="f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397" Oct 30 22:41:05.634938 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:05.634916 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397\": container with ID starting with f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397 not found: ID does not exist" containerID="f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397" Oct 30 22:41:05.635010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.634952 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397"} err="failed to get container status \"f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397\": rpc error: code = NotFound desc = could not find container \"f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397\": container with ID starting with f68b3f4fdb6ddb18c6be02b92e195e8871d172a962e024d0230f7baf70d3f397 not found: ID does not exist" Oct 30 22:41:05.678369 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:05.678311 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7706-fc6448\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" pod="opendatahub/test-curl-7706-fc6448" Oct 30 22:41:05.888651 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:05.888577 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab138fa6-0f56-41e8-a626-1dca0d59e18c" path="/var/lib/kubelet/pods/ab138fa6-0f56-41e8-a626-1dca0d59e18c/volumes" Oct 30 22:41:06.622816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:06.622778 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" event={"ID":"768d126b-8220-489e-91df-595b1bdce9b6","Type":"ContainerStarted","Data":"7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd"} Oct 30 22:41:06.622991 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:06.622884 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" event={"ID":"768d126b-8220-489e-91df-595b1bdce9b6","Type":"ContainerStarted","Data":"ba95dd2a2f99d06c484daaab5c2564735388b4fd85b204f34baa107ad4e89067"} Oct 30 22:41:06.787847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:06.787783 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podStartSLOduration=2.787770121 podStartE2EDuration="2.787770121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:06.783285256 +0000 UTC m=+3287.082695206" watchObservedRunningTime="2026-10-30 22:41:06.787770121 +0000 UTC m=+3287.087180001" Oct 30 22:41:07.144769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:07.144739 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:07.150117 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:07.150093 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:07.631165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:07.631135 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:07.632105 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:07.632081 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:41:14.864847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:14.864801 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm"] Oct 30 22:41:17.911346 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:17.911277 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:41:20.471428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:20.471395 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:41:20.485563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:20.485542 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:41:22.937078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:22.937045 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv"] Oct 30 22:41:22.937670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:22.937239 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" containerID="cri-o://b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4" gracePeriod=30 Oct 30 22:41:26.093303 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.093265 4991 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:41:26.093711 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.093343 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:41:26.488000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.487966 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6"] Oct 30 22:41:26.493864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.493842 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.549612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.549577 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6"] Oct 30 22:41:26.589822 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.589791 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cbe3a9\"/\"default-dockercfg-hdzjr\"" Oct 30 22:41:26.590041 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.589828 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cbe3a9\"/\"kube-root-ca.crt\"" Oct 30 22:41:26.590150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.589856 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cbe3a9\"/\"explicit-model-cbe3a9-kserve-self-signed-certs\"" Oct 30 22:41:26.590150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.589862 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cbe3a9\"/\"openshift-service-ca.crt\"" Oct 30 22:41:26.610402 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.610378 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-model-cache\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.610518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.610444 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-home\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.610573 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.610519 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tmp-dir\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.610612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.610569 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kt9\" (UniqueName: \"kubernetes.io/projected/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-kube-api-access-k8kt9\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.610741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.610728 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-dshm\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.610786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.610748 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tls-certs\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712006 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.711970 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-dshm\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712006 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.712007 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tls-certs\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.712034 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-model-cache\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.712089 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-home\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.712107 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tmp-dir\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.712126 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kt9\" (UniqueName: \"kubernetes.io/projected/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-kube-api-access-k8kt9\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.712579 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-home\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.712754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.712589 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tmp-dir\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.713093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.713067 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-model-cache\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.714550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.714525 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-dshm\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.714755 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.714737 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tls-certs\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.770347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.770277 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kt9\" (UniqueName: \"kubernetes.io/projected/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-kube-api-access-k8kt9\") pod \"explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:26.805971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:26.805946 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:27.190729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:27.190651 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6"] Oct 30 22:41:27.196229 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:41:27.196202 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f793f9e_c212_4fbb_b846_9d0cec55f9fb.slice/crio-a0c4e3c0536c8745f2a60ba0bedea4be04af7b558e3b3e99894f220eed35e020 WatchSource:0}: Error finding container a0c4e3c0536c8745f2a60ba0bedea4be04af7b558e3b3e99894f220eed35e020: Status 404 returned error can't find the container with id a0c4e3c0536c8745f2a60ba0bedea4be04af7b558e3b3e99894f220eed35e020 Oct 30 22:41:27.811381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:27.811343 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" event={"ID":"8f793f9e-c212-4fbb-b846-9d0cec55f9fb","Type":"ContainerStarted","Data":"30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d"} Oct 30 22:41:27.811562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:27.811391 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" event={"ID":"8f793f9e-c212-4fbb-b846-9d0cec55f9fb","Type":"ContainerStarted","Data":"a0c4e3c0536c8745f2a60ba0bedea4be04af7b558e3b3e99894f220eed35e020"} Oct 30 22:41:27.925388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:27.925307 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" podStartSLOduration=1.925290564 podStartE2EDuration="1.925290564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:27.921840821 +0000 UTC m=+3308.221250701" watchObservedRunningTime="2026-10-30 22:41:27.925290564 +0000 UTC m=+3308.224700446" Oct 30 22:41:28.231084 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.231052 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:41:28.337496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337418 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-kube-api-access-rmmvq\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-kube-api-access-rmmvq\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.337496 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337473 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-certs\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.337734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337506 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e648c983-a074-4f69-ad9c-bdf65387e7c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e648c983-a074-4f69-ad9c-bdf65387e7c1-istiod-ca-cert\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.337734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337528 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-envoy\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.337734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337562 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-token\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-token\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.337734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337610 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-podinfo\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.337734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337633 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-socket\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.337734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337688 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-credential-socket\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.338047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337873 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-certs" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:28.338047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337908 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-data\") pod \"e648c983-a074-4f69-ad9c-bdf65387e7c1\" (UID: \"e648c983-a074-4f69-ad9c-bdf65387e7c1\") " Oct 30 22:41:28.338047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.337977 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-socket" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:28.338259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.338095 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-credential-socket" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:28.338335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.338288 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-data" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:28.338585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.338561 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e648c983-a074-4f69-ad9c-bdf65387e7c1-istiod-ca-cert" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:28.338585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.338577 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.338703 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.338597 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.338703 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.338612 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.338703 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.338626 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.340175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.340140 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-kube-api-access-rmmvq" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "kube-api-access-rmmvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:28.340389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.340368 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-envoy" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:28.340534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.340510 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-token" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:28.340761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.340727 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-podinfo" pod "e648c983-a074-4f69-ad9c-bdf65387e7c1" (UID: "e648c983-a074-4f69-ad9c-bdf65387e7c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:28.439982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.439948 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmmvq\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-kube-api-access-rmmvq\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.439982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.439978 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e648c983-a074-4f69-ad9c-bdf65387e7c1-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.439982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.439988 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.440271 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.439998 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.440271 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.440006 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e648c983-a074-4f69-ad9c-bdf65387e7c1-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:28.806941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.806913 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:28.819092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.819062 4991 generic.go:356] "Generic (PLEG): container finished" podID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerID="b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4" exitCode=0 Oct 30 22:41:28.819236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.819113 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" Oct 30 22:41:28.819236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.819146 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" event={"ID":"e648c983-a074-4f69-ad9c-bdf65387e7c1","Type":"ContainerDied","Data":"b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4"} Oct 30 22:41:28.819236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.819220 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv" event={"ID":"e648c983-a074-4f69-ad9c-bdf65387e7c1","Type":"ContainerDied","Data":"dfe4f238ddb60ac1431d22c4ef109534c33b4d110110d2887b6a0b3fd4c72a38"} Oct 30 22:41:28.819415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.819241 4991 scope.go:122] "RemoveContainer" containerID="b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4" Oct 30 22:41:28.832695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.832674 4991 scope.go:122] "RemoveContainer" containerID="b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4" Oct 30 22:41:28.832973 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:28.832950 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4\": container with ID starting with b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4 not found: ID does not exist" containerID="b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4" Oct 30 22:41:28.833064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.832982 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4"} err="failed to get container status \"b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4\": rpc error: code = NotFound desc = could not find container \"b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4\": container with ID starting with b94de2e27a575b8c673b4dc008af7654ec79588c692a270c1491e3adf8bde2f4 not found: ID does not exist" Oct 30 22:41:28.976727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:28.976692 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv"] Oct 30 22:41:29.023443 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:29.023413 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wcctv"] Oct 30 22:41:29.878669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:29.878634 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" path="/var/lib/kubelet/pods/e648c983-a074-4f69-ad9c-bdf65387e7c1/volumes" Oct 30 22:41:29.894672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:29.894633 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" podUID="54e9b919-9860-4ace-9411-fc163a74b3e9" containerName="main" containerID="cri-o://3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72" gracePeriod=45 Oct 30 22:41:30.181713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.181685 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:41:30.263566 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.263539 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-dshm\") pod \"54e9b919-9860-4ace-9411-fc163a74b3e9\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " Oct 30 22:41:30.263749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.263600 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e9b919-9860-4ace-9411-fc163a74b3e9-kube-api-access-lmz64\" (UniqueName: \"kubernetes.io/projected/54e9b919-9860-4ace-9411-fc163a74b3e9-kube-api-access-lmz64\") pod \"54e9b919-9860-4ace-9411-fc163a74b3e9\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " Oct 30 22:41:30.263749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.263633 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-tmp-dir\") pod \"54e9b919-9860-4ace-9411-fc163a74b3e9\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " Oct 30 22:41:30.263749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.263731 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e9b919-9860-4ace-9411-fc163a74b3e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/54e9b919-9860-4ace-9411-fc163a74b3e9-tls-certs\") pod \"54e9b919-9860-4ace-9411-fc163a74b3e9\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " Oct 30 22:41:30.263921 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.263801 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-model-cache\") pod \"54e9b919-9860-4ace-9411-fc163a74b3e9\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " Oct 30 22:41:30.263921 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.263840 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-home\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-home\") pod \"54e9b919-9860-4ace-9411-fc163a74b3e9\" (UID: \"54e9b919-9860-4ace-9411-fc163a74b3e9\") " Oct 30 22:41:30.264034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.264004 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-tmp-dir" pod "54e9b919-9860-4ace-9411-fc163a74b3e9" (UID: "54e9b919-9860-4ace-9411-fc163a74b3e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.264115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.264075 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-model-cache" pod "54e9b919-9860-4ace-9411-fc163a74b3e9" (UID: "54e9b919-9860-4ace-9411-fc163a74b3e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.264219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.264200 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-home" pod "54e9b919-9860-4ace-9411-fc163a74b3e9" (UID: "54e9b919-9860-4ace-9411-fc163a74b3e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.264508 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.264475 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.264667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.264520 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.264667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.264532 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.266502 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.266187 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-dshm" pod "54e9b919-9860-4ace-9411-fc163a74b3e9" (UID: "54e9b919-9860-4ace-9411-fc163a74b3e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.269761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.269731 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e9b919-9860-4ace-9411-fc163a74b3e9-tls-certs" pod "54e9b919-9860-4ace-9411-fc163a74b3e9" (UID: "54e9b919-9860-4ace-9411-fc163a74b3e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:30.270286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.270263 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e9b919-9860-4ace-9411-fc163a74b3e9-kube-api-access-lmz64" pod "54e9b919-9860-4ace-9411-fc163a74b3e9" (UID: "54e9b919-9860-4ace-9411-fc163a74b3e9"). InnerVolumeSpecName "kube-api-access-lmz64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.369543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.369482 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmz64\" (UniqueName: \"kubernetes.io/projected/54e9b919-9860-4ace-9411-fc163a74b3e9-kube-api-access-lmz64\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.369543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.369528 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e9b919-9860-4ace-9411-fc163a74b3e9-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.369543 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.369544 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e9b919-9860-4ace-9411-fc163a74b3e9-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.837525 4991 generic.go:356] "Generic (PLEG): container finished" podID="54e9b919-9860-4ace-9411-fc163a74b3e9" containerID="3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72" exitCode=0 Oct 30 22:41:30.837769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.837580 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" Oct 30 22:41:30.837769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.837603 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" event={"ID":"54e9b919-9860-4ace-9411-fc163a74b3e9","Type":"ContainerDied","Data":"3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72"} Oct 30 22:41:30.837769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.837651 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm" event={"ID":"54e9b919-9860-4ace-9411-fc163a74b3e9","Type":"ContainerDied","Data":"6b000116a66ec8fe2bc7f40e8a9d8cbe7cde78c231a240581a816d4ec072b229"} Oct 30 22:41:30.837769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.837675 4991 scope.go:122] "RemoveContainer" containerID="3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72" Oct 30 22:41:30.849116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.849096 4991 scope.go:122] "RemoveContainer" containerID="3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72" Oct 30 22:41:30.849449 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:30.849427 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72\": container with ID starting with 3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72 not found: ID does not exist" containerID="3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72" Oct 30 22:41:30.849545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.849457 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72"} err="failed to get container status \"3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72\": rpc error: code = NotFound desc = could not find container \"3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72\": container with ID starting with 3eb31d7875f6588604a71830560d51a8f6e5aca524229b158129c7c9db581a72 not found: ID does not exist" Oct 30 22:41:30.994381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:30.994350 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm"] Oct 30 22:41:31.041208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:31.041184 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-68f80c/auth-test-model-68f80c-kserve-bbf78c949-49wbm"] Oct 30 22:41:31.874547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:31.874511 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e9b919-9860-4ace-9411-fc163a74b3e9" path="/var/lib/kubelet/pods/54e9b919-9860-4ace-9411-fc163a74b3e9/volumes" Oct 30 22:41:36.807050 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:36.807014 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:36.819477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:36.819452 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:36.893500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:36.893478 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:41:47.910633 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:41:47.910596 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:41:53.188083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:41:53.188049 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn"] Oct 30 22:42:08.208369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.208312 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" podUID="38eb7556-0637-41b8-9d8e-c6d6773f76ca" containerName="main" containerID="cri-o://16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286" gracePeriod=45 Oct 30 22:42:08.473621 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.473600 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:42:08.640142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640101 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-home\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-home\") pod \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " Oct 30 22:42:08.640342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640154 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38eb7556-0637-41b8-9d8e-c6d6773f76ca-kube-api-access-nb6lg\" (UniqueName: \"kubernetes.io/projected/38eb7556-0637-41b8-9d8e-c6d6773f76ca-kube-api-access-nb6lg\") pod \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " Oct 30 22:42:08.640342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640243 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-dshm\") pod \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " Oct 30 22:42:08.640342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640293 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tls-certs\") pod \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " Oct 30 22:42:08.640523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640436 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-model-cache\") pod \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " Oct 30 22:42:08.640523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640453 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-home" pod "38eb7556-0637-41b8-9d8e-c6d6773f76ca" (UID: "38eb7556-0637-41b8-9d8e-c6d6773f76ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:08.640523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640496 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tmp-dir\") pod \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\" (UID: \"38eb7556-0637-41b8-9d8e-c6d6773f76ca\") " Oct 30 22:42:08.640698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640679 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-model-cache" pod "38eb7556-0637-41b8-9d8e-c6d6773f76ca" (UID: "38eb7556-0637-41b8-9d8e-c6d6773f76ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:08.640900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.640874 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tmp-dir" pod "38eb7556-0637-41b8-9d8e-c6d6773f76ca" (UID: "38eb7556-0637-41b8-9d8e-c6d6773f76ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:08.641067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.641047 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:08.641139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.641070 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:08.641139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.641080 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:08.642907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.642886 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-dshm" pod "38eb7556-0637-41b8-9d8e-c6d6773f76ca" (UID: "38eb7556-0637-41b8-9d8e-c6d6773f76ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:08.643008 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.642927 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tls-certs" pod "38eb7556-0637-41b8-9d8e-c6d6773f76ca" (UID: "38eb7556-0637-41b8-9d8e-c6d6773f76ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:08.643008 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.642933 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38eb7556-0637-41b8-9d8e-c6d6773f76ca-kube-api-access-nb6lg" pod "38eb7556-0637-41b8-9d8e-c6d6773f76ca" (UID: "38eb7556-0637-41b8-9d8e-c6d6773f76ca"). InnerVolumeSpecName "kube-api-access-nb6lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:08.742298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.742239 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6lg\" (UniqueName: \"kubernetes.io/projected/38eb7556-0637-41b8-9d8e-c6d6773f76ca-kube-api-access-nb6lg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:08.742298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.742263 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38eb7556-0637-41b8-9d8e-c6d6773f76ca-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:08.742298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:08.742275 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb7556-0637-41b8-9d8e-c6d6773f76ca-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:09.107147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.107048 4991 generic.go:356] "Generic (PLEG): container finished" podID="38eb7556-0637-41b8-9d8e-c6d6773f76ca" containerID="16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286" exitCode=0 Oct 30 22:42:09.107147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.107123 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" Oct 30 22:42:09.107147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.107134 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" event={"ID":"38eb7556-0637-41b8-9d8e-c6d6773f76ca","Type":"ContainerDied","Data":"16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286"} Oct 30 22:42:09.107430 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.107177 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn" event={"ID":"38eb7556-0637-41b8-9d8e-c6d6773f76ca","Type":"ContainerDied","Data":"782c64de7ca4d87565049ac69c17b92b663e7318f1dc4dcfc012da473f692d77"} Oct 30 22:42:09.107430 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.107193 4991 scope.go:122] "RemoveContainer" containerID="16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286" Oct 30 22:42:09.118331 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.118302 4991 scope.go:122] "RemoveContainer" containerID="16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286" Oct 30 22:42:09.118614 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:42:09.118594 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286\": container with ID starting with 16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286 not found: ID does not exist" containerID="16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286" Oct 30 22:42:09.118687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.118625 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286"} err="failed to get container status \"16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286\": rpc error: code = NotFound desc = could not find container \"16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286\": container with ID starting with 16278ccd5322ecaf89e0561973f1d28d0cb64be74db4952d73567a49a305b286 not found: ID does not exist" Oct 30 22:42:09.267788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.267753 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn"] Oct 30 22:42:09.312542 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.312508 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f39eb7/auth-test-model-f39eb7-kserve-7d5b5589-8fvpn"] Oct 30 22:42:09.876312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:09.876280 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38eb7556-0637-41b8-9d8e-c6d6773f76ca" path="/var/lib/kubelet/pods/38eb7556-0637-41b8-9d8e-c6d6773f76ca/volumes" Oct 30 22:42:17.911158 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:42:17.911119 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:42:25.781559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.781527 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms"] Oct 30 22:42:25.782122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782106 4991 cpu_manager.go:414] "containerMap: removing container" podUID="54e9b919-9860-4ace-9411-fc163a74b3e9" containerName="main" Oct 30 22:42:25.782169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782122 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e9b919-9860-4ace-9411-fc163a74b3e9" containerName="main" Oct 30 22:42:25.782169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782137 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" Oct 30 22:42:25.782169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782142 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" Oct 30 22:42:25.782169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782169 4991 cpu_manager.go:414] "containerMap: removing container" podUID="38eb7556-0637-41b8-9d8e-c6d6773f76ca" containerName="main" Oct 30 22:42:25.782299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782174 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38eb7556-0637-41b8-9d8e-c6d6773f76ca" containerName="main" Oct 30 22:42:25.782299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782263 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e9b919-9860-4ace-9411-fc163a74b3e9" containerName="main" Oct 30 22:42:25.782299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782272 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e648c983-a074-4f69-ad9c-bdf65387e7c1" containerName="istio-proxy" Oct 30 22:42:25.782299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.782283 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="38eb7556-0637-41b8-9d8e-c6d6773f76ca" containerName="main" Oct 30 22:42:25.787525 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.787498 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.842690 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.838411 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b829bc-openshift-default-dockercfg-zmkzp\"" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.919762 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istiod-ca-cert\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.919830 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-credential-socket\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.919861 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-envoy\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.919995 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-certs\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.920057 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gwn\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-kube-api-access-65gwn\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.920088 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-socket\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.920133 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-podinfo\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.920763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.920155 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-token\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.951467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.921474 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-data\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:25.951467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:25.946986 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms"] Oct 30 22:42:26.023089 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023055 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-certs\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023125 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gwn\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-kube-api-access-65gwn\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023151 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-socket\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023183 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-podinfo\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023208 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-token\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023264 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-data\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023352 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istiod-ca-cert\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023397 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-credential-socket\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023423 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-envoy\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023534 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-certs\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023800 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023776 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-data\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023896 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-credential-socket\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.023965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.023922 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-socket\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.024289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.024267 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istiod-ca-cert\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.025798 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.025769 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-envoy\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.025909 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.025834 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-podinfo\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.082308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.082243 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-token\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.082308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.082278 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gwn\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-kube-api-access-65gwn\") pod \"e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.100953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.100927 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:26.558829 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.558791 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms"] Oct 30 22:42:26.564750 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:42:26.564720 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad2290c_3dae_4d9a_b985_8820a0611b5c.slice/crio-dc5572ca6be86890153cbd89f38ee170ddee8713c165234dcb4ce8cca9d63cd7 WatchSource:0}: Error finding container dc5572ca6be86890153cbd89f38ee170ddee8713c165234dcb4ce8cca9d63cd7: Status 404 returned error can't find the container with id dc5572ca6be86890153cbd89f38ee170ddee8713c165234dcb4ce8cca9d63cd7 Oct 30 22:42:26.567221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.567184 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:26.567335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.567262 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:26.567335 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:26.567291 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:27.243824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:27.243786 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" event={"ID":"2ad2290c-3dae-4d9a-b985-8820a0611b5c","Type":"ContainerStarted","Data":"15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8"} Oct 30 22:42:27.244298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:27.243845 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" event={"ID":"2ad2290c-3dae-4d9a-b985-8820a0611b5c","Type":"ContainerStarted","Data":"dc5572ca6be86890153cbd89f38ee170ddee8713c165234dcb4ce8cca9d63cd7"} Oct 30 22:42:27.360388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:27.360335 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podStartSLOduration=2.360305546 podStartE2EDuration="2.360305546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:27.358365442 +0000 UTC m=+3367.657775322" watchObservedRunningTime="2026-10-30 22:42:27.360305546 +0000 UTC m=+3367.659715426" Oct 30 22:42:28.101808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:28.101772 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:28.107864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:28.107836 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:28.250837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:28.250773 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:28.251820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:28.251798 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:42:28.539819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:28.539782 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6"] Oct 30 22:42:43.561937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.561894 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" podUID="8f793f9e-c212-4fbb-b846-9d0cec55f9fb" containerName="main" containerID="cri-o://30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d" gracePeriod=45 Oct 30 22:42:43.891569 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.891542 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:42:43.985206 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985176 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-kube-api-access-k8kt9\" (UniqueName: \"kubernetes.io/projected/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-kube-api-access-k8kt9\") pod \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " Oct 30 22:42:43.985415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985338 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tls-certs\") pod \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " Oct 30 22:42:43.985415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985391 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tmp-dir\") pod \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " Oct 30 22:42:43.985546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985421 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-dshm\") pod \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " Oct 30 22:42:43.985546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985470 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-home\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-home\") pod \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " Oct 30 22:42:43.985671 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985557 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-model-cache\") pod \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\" (UID: \"8f793f9e-c212-4fbb-b846-9d0cec55f9fb\") " Oct 30 22:42:43.985756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985651 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tmp-dir" pod "8f793f9e-c212-4fbb-b846-9d0cec55f9fb" (UID: "8f793f9e-c212-4fbb-b846-9d0cec55f9fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:43.986034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985974 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-home" pod "8f793f9e-c212-4fbb-b846-9d0cec55f9fb" (UID: "8f793f9e-c212-4fbb-b846-9d0cec55f9fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:43.986034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.985985 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-model-cache" pod "8f793f9e-c212-4fbb-b846-9d0cec55f9fb" (UID: "8f793f9e-c212-4fbb-b846-9d0cec55f9fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:43.986219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.986141 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:43.986219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.986161 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:43.986219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.986179 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:43.987601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.987576 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-dshm" pod "8f793f9e-c212-4fbb-b846-9d0cec55f9fb" (UID: "8f793f9e-c212-4fbb-b846-9d0cec55f9fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:43.987691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.987600 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tls-certs" pod "8f793f9e-c212-4fbb-b846-9d0cec55f9fb" (UID: "8f793f9e-c212-4fbb-b846-9d0cec55f9fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:43.987735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:43.987708 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-kube-api-access-k8kt9" pod "8f793f9e-c212-4fbb-b846-9d0cec55f9fb" (UID: "8f793f9e-c212-4fbb-b846-9d0cec55f9fb"). InnerVolumeSpecName "kube-api-access-k8kt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:44.087184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.087148 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kt9\" (UniqueName: \"kubernetes.io/projected/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-kube-api-access-k8kt9\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:44.087184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.087179 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:44.087184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.087190 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f793f9e-c212-4fbb-b846-9d0cec55f9fb-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:44.383900 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.383865 4991 generic.go:356] "Generic (PLEG): container finished" podID="8f793f9e-c212-4fbb-b846-9d0cec55f9fb" containerID="30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d" exitCode=0 Oct 30 22:42:44.384091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.383923 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" Oct 30 22:42:44.384091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.383951 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" event={"ID":"8f793f9e-c212-4fbb-b846-9d0cec55f9fb","Type":"ContainerDied","Data":"30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d"} Oct 30 22:42:44.384091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.384000 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" event={"ID":"8f793f9e-c212-4fbb-b846-9d0cec55f9fb","Type":"ContainerDied","Data":"a0c4e3c0536c8745f2a60ba0bedea4be04af7b558e3b3e99894f220eed35e020"} Oct 30 22:42:44.384091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.384026 4991 scope.go:122] "RemoveContainer" containerID="30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d" Oct 30 22:42:44.395331 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.395304 4991 scope.go:122] "RemoveContainer" containerID="30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d" Oct 30 22:42:44.395675 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:42:44.395655 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d\": container with ID starting with 30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d not found: ID does not exist" containerID="30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d" Oct 30 22:42:44.395743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.395683 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d"} err="failed to get container status \"30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d\": rpc error: code = NotFound desc = could not find container \"30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d\": container with ID starting with 30582f07b4499a433969280cbdab60bf426c1c82696d9824ee89e9ce9982ce1d not found: ID does not exist" Oct 30 22:42:44.552507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.552467 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6"] Oct 30 22:42:44.557554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.557524 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6"] Oct 30 22:42:44.882818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:44.882782 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-cbe3a9/explicit-model-cbe3a9-kserve-68c6cf6d69-fr2f6" podUID="8f793f9e-c212-4fbb-b846-9d0cec55f9fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": context deadline exceeded" Oct 30 22:42:45.709482 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:45.709452 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk"] Oct 30 22:42:45.890215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:45.889550 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f793f9e-c212-4fbb-b846-9d0cec55f9fb" path="/var/lib/kubelet/pods/8f793f9e-c212-4fbb-b846-9d0cec55f9fb/volumes" Oct 30 22:42:47.912897 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:42:47.912847 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:42:50.742289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.742256 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls"] Oct 30 22:42:50.743230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.743207 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8f793f9e-c212-4fbb-b846-9d0cec55f9fb" containerName="main" Oct 30 22:42:50.743277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.743234 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f793f9e-c212-4fbb-b846-9d0cec55f9fb" containerName="main" Oct 30 22:42:50.743385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.743373 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f793f9e-c212-4fbb-b846-9d0cec55f9fb" containerName="main" Oct 30 22:42:50.746276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.746259 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.870830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.870751 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.870830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.870800 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ab5024-b7a9-496b-951d-ea388be139c5-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.871019 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.870964 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.871134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.871030 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxxng\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-kube-api-access-gxxng\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.871216 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.871198 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.871280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.871250 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.871417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.871396 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.871479 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.871440 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.871535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.871508 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.961008 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.959918 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls"] Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975202 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975271 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975371 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975406 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975446 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975536 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975573 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ab5024-b7a9-496b-951d-ea388be139c5-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975631 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975683 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxxng\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-kube-api-access-gxxng\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.976145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.975812 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.977503 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.977475 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.978353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.978120 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ab5024-b7a9-496b-951d-ea388be139c5-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.978353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.978264 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.978353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.978334 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.980176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.980153 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:50.981681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:50.981657 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:51.035353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.035311 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:51.042556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.042532 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxxng\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-kube-api-access-gxxng\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:51.062275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.062250 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:51.439971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.439885 4991 generic.go:356] "Generic (PLEG): container finished" podID="768d126b-8220-489e-91df-595b1bdce9b6" containerID="7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd" exitCode=0 Oct 30 22:42:51.440126 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.439967 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" event={"ID":"768d126b-8220-489e-91df-595b1bdce9b6","Type":"ContainerDied","Data":"7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd"} Oct 30 22:42:51.440535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.440517 4991 scope.go:122] "RemoveContainer" containerID="7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd" Oct 30 22:42:51.454687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.454642 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:51.454842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.454738 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:51.454842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.454785 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:51.571041 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.570991 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls"] Oct 30 22:42:51.616374 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:42:51.612719 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ab5024_b7a9_496b_951d_ea388be139c5.slice/crio-5b451679276fb293a4c0ff8ff508a37396ad84df1a7552ad9075e47c1504a7ef WatchSource:0}: Error finding container 5b451679276fb293a4c0ff8ff508a37396ad84df1a7552ad9075e47c1504a7ef: Status 404 returned error can't find the container with id 5b451679276fb293a4c0ff8ff508a37396ad84df1a7552ad9075e47c1504a7ef Oct 30 22:42:51.616638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.616374 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:51.616638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.616464 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:51.616638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:51.616510 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:42:52.479345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:52.471921 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" event={"ID":"768d126b-8220-489e-91df-595b1bdce9b6","Type":"ContainerStarted","Data":"254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085"} Oct 30 22:42:52.481278 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:52.481240 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" event={"ID":"c0ab5024-b7a9-496b-951d-ea388be139c5","Type":"ContainerStarted","Data":"4501e2c7ebbdf35d8d158f8562169e90a80455912775b44d1327969831a8e792"} Oct 30 22:42:52.481530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:52.481509 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" event={"ID":"c0ab5024-b7a9-496b-951d-ea388be139c5","Type":"ContainerStarted","Data":"5b451679276fb293a4c0ff8ff508a37396ad84df1a7552ad9075e47c1504a7ef"} Oct 30 22:42:52.652343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:52.645492 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:42:52.715049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:52.714988 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podStartSLOduration=2.714974778 podStartE2EDuration="2.714974778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:52.710546299 +0000 UTC m=+3393.009956180" watchObservedRunningTime="2026-10-30 22:42:52.714974778 +0000 UTC m=+3393.014384658" Oct 30 22:42:53.065730 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:53.065693 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:53.070489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:53.070436 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:42:53.070747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:53.070716 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:42:53.145433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:53.145382 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:42:53.145715 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:53.145683 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:42:53.145911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:53.145882 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:42:54.065805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:54.065769 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:42:54.066437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:54.066393 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:42:54.148344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:54.147792 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:42:54.148344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:54.147866 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:42:55.065765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:55.065721 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:42:55.066057 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:55.066027 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:42:55.148438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:55.148402 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:42:55.148739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:55.148710 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:42:56.065351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:56.064707 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:42:56.065351 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:56.064769 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:42:56.145675 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:56.145635 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:42:56.146242 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:56.146213 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:42:57.063344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:57.062713 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:42:57.063344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:57.062779 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:42:57.145739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:57.145703 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:42:57.146342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:57.146294 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 22:42:58.062713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:58.062677 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:42:58.062893 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:58.062730 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:42:58.148988 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:58.148959 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:42:58.150067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:58.150047 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:42:59.068145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:59.068115 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:59.068547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:59.068525 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:42:59.069254 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:42:59.069234 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:43:00.761017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:00.760982 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" podUID="288c14ac-8b95-4a43-b368-e11e9a5cbaa1" containerName="main" containerID="cri-o://b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3" gracePeriod=45 Oct 30 22:43:01.021191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.021137 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:43:01.051953 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.051927 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tmp-dir\") pod \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " Oct 30 22:43:01.052088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.051972 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-model-cache\") pod \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " Oct 30 22:43:01.052088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052002 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-home\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-home\") pod \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " Oct 30 22:43:01.052088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052020 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tls-certs\" (UniqueName: \"kubernetes.io/secret/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tls-certs\") pod \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " Oct 30 22:43:01.052270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052127 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-kube-api-access-t789m\" (UniqueName: \"kubernetes.io/projected/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-kube-api-access-t789m\") pod \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " Oct 30 22:43:01.052270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052175 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-dshm\") pod \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\" (UID: \"288c14ac-8b95-4a43-b368-e11e9a5cbaa1\") " Oct 30 22:43:01.052270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052199 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-model-cache" pod "288c14ac-8b95-4a43-b368-e11e9a5cbaa1" (UID: "288c14ac-8b95-4a43-b368-e11e9a5cbaa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:01.052270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052220 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tmp-dir" pod "288c14ac-8b95-4a43-b368-e11e9a5cbaa1" (UID: "288c14ac-8b95-4a43-b368-e11e9a5cbaa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:01.052452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052294 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-home" pod "288c14ac-8b95-4a43-b368-e11e9a5cbaa1" (UID: "288c14ac-8b95-4a43-b368-e11e9a5cbaa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:01.052644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052623 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:01.052644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052644 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:01.052821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.052653 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:01.054285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.054255 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-dshm" pod "288c14ac-8b95-4a43-b368-e11e9a5cbaa1" (UID: "288c14ac-8b95-4a43-b368-e11e9a5cbaa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:01.054429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.054302 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tls-certs" pod "288c14ac-8b95-4a43-b368-e11e9a5cbaa1" (UID: "288c14ac-8b95-4a43-b368-e11e9a5cbaa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:01.054429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.054366 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-kube-api-access-t789m" pod "288c14ac-8b95-4a43-b368-e11e9a5cbaa1" (UID: "288c14ac-8b95-4a43-b368-e11e9a5cbaa1"). InnerVolumeSpecName "kube-api-access-t789m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:01.153879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.153845 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:01.153879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.153878 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t789m\" (UniqueName: \"kubernetes.io/projected/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-kube-api-access-t789m\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:01.154087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.153894 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288c14ac-8b95-4a43-b368-e11e9a5cbaa1-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:01.542121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.542078 4991 generic.go:356] "Generic (PLEG): container finished" podID="288c14ac-8b95-4a43-b368-e11e9a5cbaa1" containerID="b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3" exitCode=0 Oct 30 22:43:01.542290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.542132 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" Oct 30 22:43:01.542290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.542171 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" event={"ID":"288c14ac-8b95-4a43-b368-e11e9a5cbaa1","Type":"ContainerDied","Data":"b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3"} Oct 30 22:43:01.542290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.542211 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk" event={"ID":"288c14ac-8b95-4a43-b368-e11e9a5cbaa1","Type":"ContainerDied","Data":"a66fe451f32d133eead1ddb86addcb9a0a43f896e62d7524e54ad8286c08f6ef"} Oct 30 22:43:01.542290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.542232 4991 scope.go:122] "RemoveContainer" containerID="b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3" Oct 30 22:43:01.553538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.553517 4991 scope.go:122] "RemoveContainer" containerID="b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3" Oct 30 22:43:01.554621 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:01.554595 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3\": container with ID starting with b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3 not found: ID does not exist" containerID="b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3" Oct 30 22:43:01.554709 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.554636 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3"} err="failed to get container status \"b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3\": rpc error: code = NotFound desc = could not find container \"b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3\": container with ID starting with b8c3f974d2c2496a8a68f2a5de041922736d128dc64c34467c032e963075fde3 not found: ID does not exist" Oct 30 22:43:01.705004 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.704972 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk"] Oct 30 22:43:01.750113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.750078 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-37e79b/auth-test-model-37e79b-kserve-5c5c957469-tj9tk"] Oct 30 22:43:01.875292 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:01.875217 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288c14ac-8b95-4a43-b368-e11e9a5cbaa1" path="/var/lib/kubelet/pods/288c14ac-8b95-4a43-b368-e11e9a5cbaa1/volumes" Oct 30 22:43:17.911428 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:17.911386 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:43:22.717336 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.717297 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-2a919c"] Oct 30 22:43:22.718065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.718042 4991 cpu_manager.go:414] "containerMap: removing container" podUID="288c14ac-8b95-4a43-b368-e11e9a5cbaa1" containerName="main" Oct 30 22:43:22.718194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.718067 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288c14ac-8b95-4a43-b368-e11e9a5cbaa1" containerName="main" Oct 30 22:43:22.718259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.718226 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="288c14ac-8b95-4a43-b368-e11e9a5cbaa1" containerName="main" Oct 30 22:43:22.723163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.723142 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:22.772236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.772209 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-2a919c"] Oct 30 22:43:22.814851 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.814831 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmbvp\"" Oct 30 22:43:22.817486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.817453 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxq6\" (UniqueName: \"kubernetes.io/projected/9fcd13a6-e9dc-4879-9038-42dacaa06c2b-kube-api-access-wkxq6\") pod \"test-curl-7700-2a919c\" (UID: \"9fcd13a6-e9dc-4879-9038-42dacaa06c2b\") " pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:22.919038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.919010 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxq6\" (UniqueName: \"kubernetes.io/projected/9fcd13a6-e9dc-4879-9038-42dacaa06c2b-kube-api-access-wkxq6\") pod \"test-curl-7700-2a919c\" (UID: \"9fcd13a6-e9dc-4879-9038-42dacaa06c2b\") " pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:22.977223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:22.977199 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxq6\" (UniqueName: \"kubernetes.io/projected/9fcd13a6-e9dc-4879-9038-42dacaa06c2b-kube-api-access-wkxq6\") pod \"test-curl-7700-2a919c\" (UID: \"9fcd13a6-e9dc-4879-9038-42dacaa06c2b\") " pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:23.035402 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:23.035379 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:23.412988 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:23.412915 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-2a919c"] Oct 30 22:43:23.419218 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:43:23.419190 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fcd13a6_e9dc_4879_9038_42dacaa06c2b.slice/crio-8a73352d02cf6a738bb3609df01d81a67c0ce9cc0db6dced44ac5cfde61005a6 WatchSource:0}: Error finding container 8a73352d02cf6a738bb3609df01d81a67c0ce9cc0db6dced44ac5cfde61005a6: Status 404 returned error can't find the container with id 8a73352d02cf6a738bb3609df01d81a67c0ce9cc0db6dced44ac5cfde61005a6 Oct 30 22:43:23.716415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:23.716315 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-2a919c" event={"ID":"9fcd13a6-e9dc-4879-9038-42dacaa06c2b","Type":"ContainerStarted","Data":"8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5"} Oct 30 22:43:23.716415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:23.716372 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-2a919c" event={"ID":"9fcd13a6-e9dc-4879-9038-42dacaa06c2b","Type":"ContainerStarted","Data":"8a73352d02cf6a738bb3609df01d81a67c0ce9cc0db6dced44ac5cfde61005a6"} Oct 30 22:43:23.826721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:23.826671 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7700-2a919c" podStartSLOduration=1.826657667 podStartE2EDuration="1.826657667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:23.823784491 +0000 UTC m=+3424.123194390" watchObservedRunningTime="2026-10-30 22:43:23.826657667 +0000 UTC m=+3424.126067547" Oct 30 22:43:25.374763 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:25.374672 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-2a919c"] Oct 30 22:43:25.377277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:25.377129 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-2a919c"] Oct 30 22:43:25.729656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:25.729579 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7700-2a919c" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" containerName="test-curl-7700-2a919c" containerID="cri-o://8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5" gracePeriod=2 Oct 30 22:43:25.782162 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:25.782127 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7700-2a919c\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:27.496654 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.496618 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm"] Oct 30 22:43:27.497169 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.496888 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" podUID="d972c878-20dd-426d-add5-ac94c39d35d5" containerName="maas-api" containerID="cri-o://c2edf20ae6f3476a117d611311e2bee3ad7599e62669d7149c79849ae645de48" gracePeriod=30 Oct 30 22:43:27.754822 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.754790 4991 generic.go:356] "Generic (PLEG): container finished" podID="d972c878-20dd-426d-add5-ac94c39d35d5" containerID="c2edf20ae6f3476a117d611311e2bee3ad7599e62669d7149c79849ae645de48" exitCode=0 Oct 30 22:43:27.754954 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.754853 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" event={"ID":"d972c878-20dd-426d-add5-ac94c39d35d5","Type":"ContainerDied","Data":"c2edf20ae6f3476a117d611311e2bee3ad7599e62669d7149c79849ae645de48"} Oct 30 22:43:27.816914 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.816880 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:43:27.925355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.922353 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d972c878-20dd-426d-add5-ac94c39d35d5-kube-api-access-4nll6\" (UniqueName: \"kubernetes.io/projected/d972c878-20dd-426d-add5-ac94c39d35d5-kube-api-access-4nll6\") pod \"d972c878-20dd-426d-add5-ac94c39d35d5\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " Oct 30 22:43:27.925355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.922418 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-maas-api-tls\") pod \"d972c878-20dd-426d-add5-ac94c39d35d5\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " Oct 30 22:43:27.925355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.922531 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-metrics-certs\") pod \"d972c878-20dd-426d-add5-ac94c39d35d5\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " Oct 30 22:43:27.925355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.922587 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d972c878-20dd-426d-add5-ac94c39d35d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d972c878-20dd-426d-add5-ac94c39d35d5-openshift-service-ca\") pod \"d972c878-20dd-426d-add5-ac94c39d35d5\" (UID: \"d972c878-20dd-426d-add5-ac94c39d35d5\") " Oct 30 22:43:27.925355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.923433 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d972c878-20dd-426d-add5-ac94c39d35d5-openshift-service-ca" pod "d972c878-20dd-426d-add5-ac94c39d35d5" (UID: "d972c878-20dd-426d-add5-ac94c39d35d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:27.925355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.924101 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d972c878-20dd-426d-add5-ac94c39d35d5-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:27.927999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.926598 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-metrics-certs" pod "d972c878-20dd-426d-add5-ac94c39d35d5" (UID: "d972c878-20dd-426d-add5-ac94c39d35d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:27.927999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.926706 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d972c878-20dd-426d-add5-ac94c39d35d5-kube-api-access-4nll6" pod "d972c878-20dd-426d-add5-ac94c39d35d5" (UID: "d972c878-20dd-426d-add5-ac94c39d35d5"). InnerVolumeSpecName "kube-api-access-4nll6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:27.927999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:27.926846 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-maas-api-tls" pod "d972c878-20dd-426d-add5-ac94c39d35d5" (UID: "d972c878-20dd-426d-add5-ac94c39d35d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:28.006842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.006773 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:28.025332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.025301 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcd13a6-e9dc-4879-9038-42dacaa06c2b-kube-api-access-wkxq6\" (UniqueName: \"kubernetes.io/projected/9fcd13a6-e9dc-4879-9038-42dacaa06c2b-kube-api-access-wkxq6\") pod \"9fcd13a6-e9dc-4879-9038-42dacaa06c2b\" (UID: \"9fcd13a6-e9dc-4879-9038-42dacaa06c2b\") " Oct 30 22:43:28.025847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.025824 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:28.025937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.025855 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nll6\" (UniqueName: \"kubernetes.io/projected/d972c878-20dd-426d-add5-ac94c39d35d5-kube-api-access-4nll6\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:28.025937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.025867 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d972c878-20dd-426d-add5-ac94c39d35d5-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:28.027530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.027495 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcd13a6-e9dc-4879-9038-42dacaa06c2b-kube-api-access-wkxq6" pod "9fcd13a6-e9dc-4879-9038-42dacaa06c2b" (UID: "9fcd13a6-e9dc-4879-9038-42dacaa06c2b"). InnerVolumeSpecName "kube-api-access-wkxq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:28.126872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.126841 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxq6\" (UniqueName: \"kubernetes.io/projected/9fcd13a6-e9dc-4879-9038-42dacaa06c2b-kube-api-access-wkxq6\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:28.763874 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.763843 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" Oct 30 22:43:28.764339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.763872 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm" event={"ID":"d972c878-20dd-426d-add5-ac94c39d35d5","Type":"ContainerDied","Data":"975eb3f1a10bf4ba7403394ba779c0fe77ba0850848275c60c490d5ac54b25e4"} Oct 30 22:43:28.764339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.763942 4991 scope.go:122] "RemoveContainer" containerID="c2edf20ae6f3476a117d611311e2bee3ad7599e62669d7149c79849ae645de48" Oct 30 22:43:28.765624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.765594 4991 generic.go:356] "Generic (PLEG): container finished" podID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" containerID="8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5" exitCode=137 Oct 30 22:43:28.765754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.765638 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:28.779181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.779159 4991 scope.go:122] "RemoveContainer" containerID="8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5" Oct 30 22:43:28.788843 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.788824 4991 scope.go:122] "RemoveContainer" containerID="8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5" Oct 30 22:43:28.789105 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:28.789087 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5\": container with ID starting with 8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5 not found: ID does not exist" containerID="8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5" Oct 30 22:43:28.789170 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.789113 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5"} err="failed to get container status \"8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5\": rpc error: code = NotFound desc = could not find container \"8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5\": container with ID starting with 8c4bee3e005fda00592a32e7d135b759e0abb24827e27100c58c447d624b08f5 not found: ID does not exist" Oct 30 22:43:28.927890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.927856 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm"] Oct 30 22:43:28.974580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:28.974547 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cbe3a9-777b7dc8f6-vf5sm"] Oct 30 22:43:29.018637 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:29.018601 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7700-2a919c\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:29.874890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:29.874856 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" path="/var/lib/kubelet/pods/9fcd13a6-e9dc-4879-9038-42dacaa06c2b/volumes" Oct 30 22:43:29.875341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:29.875310 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d972c878-20dd-426d-add5-ac94c39d35d5" path="/var/lib/kubelet/pods/d972c878-20dd-426d-add5-ac94c39d35d5/volumes" Oct 30 22:43:29.920397 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:29.920365 4991 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7700-2a919c\" is forbidden: User \"system:node:ip-10-0-137-17.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-17.us-west-2.compute.internal' and this object" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" pod="opendatahub/test-curl-7700-2a919c" Oct 30 22:43:33.833982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:33.833949 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl"] Oct 30 22:43:36.107859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.107823 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh"] Oct 30 22:43:36.108589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.108567 4991 cpu_manager.go:414] "containerMap: removing container" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" containerName="test-curl-7700-2a919c" Oct 30 22:43:36.108707 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.108590 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" containerName="test-curl-7700-2a919c" Oct 30 22:43:36.108707 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.108612 4991 cpu_manager.go:414] "containerMap: removing container" podUID="d972c878-20dd-426d-add5-ac94c39d35d5" containerName="maas-api" Oct 30 22:43:36.108707 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.108619 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d972c878-20dd-426d-add5-ac94c39d35d5" containerName="maas-api" Oct 30 22:43:36.108872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.108821 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcd13a6-e9dc-4879-9038-42dacaa06c2b" containerName="test-curl-7700-2a919c" Oct 30 22:43:36.108872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.108837 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="d972c878-20dd-426d-add5-ac94c39d35d5" containerName="maas-api" Oct 30 22:43:36.113934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.113912 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226192 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226149 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226211 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226290 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226412 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226466 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226490 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226518 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226542 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.226601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.226584 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2gg\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-kube-api-access-5t2gg\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.311761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.311727 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh"] Oct 30 22:43:36.328190 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328154 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328198 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328230 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328255 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328299 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2gg\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-kube-api-access-5t2gg\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328416 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328463 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328550 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328645 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328696 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.328975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.328926 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.329506 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.329482 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.329809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.329784 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.329942 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.329921 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.331965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.331942 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.332679 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.332656 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.386073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.386003 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.386226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.386208 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2gg\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-kube-api-access-5t2gg\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.429082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.429053 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:36.871557 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.871516 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh"] Oct 30 22:43:36.876101 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:43:36.876064 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e6ac07_556a_4cf1_a45d_4a6a02c9805b.slice/crio-e2dcbf22a09f8d788a58ba820d9cd9160436f0522f0189b22f7b081fabb3099a WatchSource:0}: Error finding container e2dcbf22a09f8d788a58ba820d9cd9160436f0522f0189b22f7b081fabb3099a: Status 404 returned error can't find the container with id e2dcbf22a09f8d788a58ba820d9cd9160436f0522f0189b22f7b081fabb3099a Oct 30 22:43:36.878431 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.878395 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:36.878534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.878477 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:36.878534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:36.878511 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:37.431475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:37.431442 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t"] Oct 30 22:43:37.432168 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:37.432135 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" containerID="cri-o://bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae" gracePeriod=30 Oct 30 22:43:37.853291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:37.853250 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" event={"ID":"26e6ac07-556a-4cf1-a45d-4a6a02c9805b","Type":"ContainerStarted","Data":"70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e"} Oct 30 22:43:37.853628 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:37.853604 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" event={"ID":"26e6ac07-556a-4cf1-a45d-4a6a02c9805b","Type":"ContainerStarted","Data":"e2dcbf22a09f8d788a58ba820d9cd9160436f0522f0189b22f7b081fabb3099a"} Oct 30 22:43:37.969104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:37.969049 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podStartSLOduration=1.969035069 podStartE2EDuration="1.969035069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:37.96683259 +0000 UTC m=+3438.266242480" watchObservedRunningTime="2026-10-30 22:43:37.969035069 +0000 UTC m=+3438.268445027" Oct 30 22:43:38.434429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.434385 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:38.442488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.442451 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:43:38.442746 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.442705 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 22:43:38.674963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.674926 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk"] Oct 30 22:43:38.681994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.681966 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.810629 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.810593 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-6587ea-openshift-default-dockercfg-9lbh4\"" Oct 30 22:43:38.812998 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.812963 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk"] Oct 30 22:43:38.829378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.828936 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-credential-socket\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.829378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829015 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istiod-ca-cert\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.829378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829052 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-envoy\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.829378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829124 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-data\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.829378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829189 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmwz\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-kube-api-access-9xmwz\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.829378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829237 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-podinfo\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.830740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829391 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-certs\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.830740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829424 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-token\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.830740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.829453 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-socket\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.964839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.964731 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-data\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.965951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.965927 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmwz\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-kube-api-access-9xmwz\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.966497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.966444 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-podinfo\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.967122 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.967102 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-certs\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.968020 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.967999 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-token\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.968468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.968448 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-socket\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.969047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.969025 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-credential-socket\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.969523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.969506 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istiod-ca-cert\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.970312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.970292 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-envoy\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.970312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.965829 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-data\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.970312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.969448 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-credential-socket\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.970312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.967532 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-certs\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.970312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.970214 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istiod-ca-cert\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.970312 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.968839 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-socket\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.992675 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.992630 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-podinfo\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:38.993189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:38.993140 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-envoy\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:39.075055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.059079 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmwz\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-kube-api-access-9xmwz\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:39.075055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.072851 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-token\") pod \"e2e-noten-6587ea-openshift-default-549c57899c-xv7qk\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:39.315788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.315753 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:39.463369 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.463313 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:43:39.463765 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.463741 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 22:43:39.799047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.798992 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk"] Oct 30 22:43:39.804860 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:43:39.804832 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cd367f_a2ba_47a7_9a2f_2fd52b5b0405.slice/crio-7ebd944c9d33f9a1f6c29ec7bc5a9f0c2f530d06bc51781bcad3de5975bf233b WatchSource:0}: Error finding container 7ebd944c9d33f9a1f6c29ec7bc5a9f0c2f530d06bc51781bcad3de5975bf233b: Status 404 returned error can't find the container with id 7ebd944c9d33f9a1f6c29ec7bc5a9f0c2f530d06bc51781bcad3de5975bf233b Oct 30 22:43:39.807634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.807595 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:39.807739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.807684 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:39.807739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.807726 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:39.967172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.967134 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" event={"ID":"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405","Type":"ContainerStarted","Data":"323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab"} Oct 30 22:43:39.967372 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:39.967192 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" event={"ID":"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405","Type":"ContainerStarted","Data":"7ebd944c9d33f9a1f6c29ec7bc5a9f0c2f530d06bc51781bcad3de5975bf233b"} Oct 30 22:43:40.085416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:40.085302 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" podStartSLOduration=2.085288561 podStartE2EDuration="2.085288561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:40.083346481 +0000 UTC m=+3440.382756354" watchObservedRunningTime="2026-10-30 22:43:40.085288561 +0000 UTC m=+3440.384698440" Oct 30 22:43:40.316735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:40.316699 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:40.317835 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:40.317810 4991 patch_prober.go:29] interesting pod/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 22:43:40.317904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:40.317870 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" podUID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 22:43:40.434846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:40.434759 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:43:40.435121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:40.435080 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 22:43:41.325256 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:41.325222 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:41.437392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:41.437356 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:43:41.437659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:41.437630 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 22:43:41.982741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:41.982698 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:41.983568 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:41.983541 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:43:42.434775 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.434686 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:43:42.435364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.435310 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 22:43:42.757042 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.757017 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:43:42.862390 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862353 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-certs\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862435 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67663820-6166-4317-a27d-83bd8d2e7843-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67663820-6166-4317-a27d-83bd8d2e7843-istio-podinfo\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862480 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-envoy\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862531 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67663820-6166-4317-a27d-83bd8d2e7843-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67663820-6166-4317-a27d-83bd8d2e7843-istiod-ca-cert\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862581 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-credential-socket\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862592 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-certs" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:42.862751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862635 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-kube-api-access-6gfds\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-kube-api-access-6gfds\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862674 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-istio-token\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-istio-token\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862756 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-data\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862814 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-socket\") pod \"67663820-6166-4317-a27d-83bd8d2e7843\" (UID: \"67663820-6166-4317-a27d-83bd8d2e7843\") " Oct 30 22:43:42.862941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862851 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67663820-6166-4317-a27d-83bd8d2e7843-istiod-ca-cert" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:42.862941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.862864 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-credential-socket" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:42.863129 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.863090 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-data" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:42.863176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.863154 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-socket" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:42.863511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.863416 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67663820-6166-4317-a27d-83bd8d2e7843-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.863511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.863443 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.863511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.863459 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.863511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.863474 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.863511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.863488 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.864966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.864943 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-kube-api-access-6gfds" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "kube-api-access-6gfds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:42.865092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.865065 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-envoy" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:42.865179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.865162 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67663820-6166-4317-a27d-83bd8d2e7843-istio-podinfo" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:42.865406 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.865387 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-istio-token" pod "67663820-6166-4317-a27d-83bd8d2e7843" (UID: "67663820-6166-4317-a27d-83bd8d2e7843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:42.965263 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.965183 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gfds\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-kube-api-access-6gfds\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.965422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.965361 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67663820-6166-4317-a27d-83bd8d2e7843-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.965422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.965375 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67663820-6166-4317-a27d-83bd8d2e7843-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.965422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.965385 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67663820-6166-4317-a27d-83bd8d2e7843-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:42.991526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.991491 4991 generic.go:356] "Generic (PLEG): container finished" podID="67663820-6166-4317-a27d-83bd8d2e7843" containerID="bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae" exitCode=0 Oct 30 22:43:42.991680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.991561 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" Oct 30 22:43:42.991680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.991571 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" event={"ID":"67663820-6166-4317-a27d-83bd8d2e7843","Type":"ContainerDied","Data":"bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae"} Oct 30 22:43:42.991680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.991625 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t" event={"ID":"67663820-6166-4317-a27d-83bd8d2e7843","Type":"ContainerDied","Data":"f6d6f57f2641209a3fda7b48583c51651cd41986b8089eaa2fc754e8920a8884"} Oct 30 22:43:42.991680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:42.991646 4991 scope.go:122] "RemoveContainer" containerID="bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae" Oct 30 22:43:43.002811 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.002791 4991 scope.go:122] "RemoveContainer" containerID="aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7" Oct 30 22:43:43.012524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.012507 4991 scope.go:122] "RemoveContainer" containerID="bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae" Oct 30 22:43:43.012788 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:43.012770 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae\": container with ID starting with bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae not found: ID does not exist" containerID="bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae" Oct 30 22:43:43.012846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.012797 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae"} err="failed to get container status \"bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae\": rpc error: code = NotFound desc = could not find container \"bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae\": container with ID starting with bf59b6eef4e55f1ff501e4b9b5e946f8d499874c400d1d40419abbc082762aae not found: ID does not exist" Oct 30 22:43:43.012846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.012813 4991 scope.go:122] "RemoveContainer" containerID="aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7" Oct 30 22:43:43.013005 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:43.012989 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7\": container with ID starting with aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7 not found: ID does not exist" containerID="aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7" Oct 30 22:43:43.013046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.013009 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7"} err="failed to get container status \"aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7\": rpc error: code = NotFound desc = could not find container \"aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7\": container with ID starting with aff74e5d4456e46f8ee2ff5c537774132343efbba31fa5738b21cd750d23d8a7 not found: ID does not exist" Oct 30 22:43:43.156970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.156936 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t"] Oct 30 22:43:43.163275 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.163249 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-cbe3a9-openshift-default-687964c589-7dw2t"] Oct 30 22:43:43.429561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.429530 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:43:43.429728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.429581 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 22:43:43.878507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:43.878468 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67663820-6166-4317-a27d-83bd8d2e7843" path="/var/lib/kubelet/pods/67663820-6166-4317-a27d-83bd8d2e7843/volumes" Oct 30 22:43:44.102159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.102125 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk"] Oct 30 22:43:44.102878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.102851 4991 cpu_manager.go:414] "containerMap: removing container" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" Oct 30 22:43:44.102878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.102876 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" Oct 30 22:43:44.103032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.102914 4991 cpu_manager.go:414] "containerMap: removing container" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" Oct 30 22:43:44.103032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.102920 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" Oct 30 22:43:44.103032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.103014 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" Oct 30 22:43:44.103032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.103028 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="67663820-6166-4317-a27d-83bd8d2e7843" containerName="istio-proxy" Oct 30 22:43:44.108063 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.108042 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.182236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.182157 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-metrics-certs\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.182236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.182224 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f946456d-2550-4e3f-adce-ff9fe63789cb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.182487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.182374 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.182487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.182419 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvk66\" (UniqueName: \"kubernetes.io/projected/f946456d-2550-4e3f-adce-ff9fe63789cb-kube-api-access-jvk66\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.216931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.216899 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b829bc-serving-cert\"" Oct 30 22:43:44.220515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.220491 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk"] Oct 30 22:43:44.283191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.283165 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.283345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.283210 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvk66\" (UniqueName: \"kubernetes.io/projected/f946456d-2550-4e3f-adce-ff9fe63789cb-kube-api-access-jvk66\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.283345 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:44.283299 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-serving-cert: secret "maas-api-e2e-extmod-a-b829bc-serving-cert" not found Oct 30 22:43:44.283345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.283331 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-metrics-certs\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.283466 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:44.283397 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls podName:f946456d-2550-4e3f-adce-ff9fe63789cb nodeName:}" failed. No retries permitted until 2026-10-30 22:43:44.78337481 +0000 UTC m=+3445.082784691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls") pod "maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" (UID: "f946456d-2550-4e3f-adce-ff9fe63789cb") : secret "maas-api-e2e-extmod-a-b829bc-serving-cert" not found Oct 30 22:43:44.283466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.283436 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f946456d-2550-4e3f-adce-ff9fe63789cb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.284040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.284022 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f946456d-2550-4e3f-adce-ff9fe63789cb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.285862 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.285836 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-metrics-certs\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.345000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.344974 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvk66\" (UniqueName: \"kubernetes.io/projected/f946456d-2550-4e3f-adce-ff9fe63789cb-kube-api-access-jvk66\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.434282 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.434202 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:44.434583 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.434564 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:44.435357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.435305 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:43:44.789462 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.789422 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:44.792487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:44.792455 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:45.024270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.024239 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:45.420308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.413607 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk"] Oct 30 22:43:45.644243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.644210 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l"] Oct 30 22:43:45.649791 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.649770 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.706756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.706663 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l"] Oct 30 22:43:45.709779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.709746 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae9d702-f807-4040-b5e8-e140bead935f-istiod-ca-cert\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.709942 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.709808 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-credential-socket\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.709942 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.709839 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llx47\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-kube-api-access-llx47\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.709942 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.709898 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-istio-token\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.710115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.710018 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-certs\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.710115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.710046 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-data\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.710115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.710102 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-socket\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.710265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.710125 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae9d702-f807-4040-b5e8-e140bead935f-istio-podinfo\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.710265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.710149 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-envoy\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.721438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.721408 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1b5075-openshift-default-dockercfg-kp42p\"" Oct 30 22:43:45.811155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811124 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-certs\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811159 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-data\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811186 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-socket\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811206 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae9d702-f807-4040-b5e8-e140bead935f-istio-podinfo\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811342 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811235 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-envoy\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811426 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae9d702-f807-4040-b5e8-e140bead935f-istiod-ca-cert\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811484 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-credential-socket\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811516 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx47\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-kube-api-access-llx47\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811578 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811544 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-istio-token\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811584 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-data\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811631 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-socket\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811684 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-certs\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.811946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.811823 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-credential-socket\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.812113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.812092 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae9d702-f807-4040-b5e8-e140bead935f-istiod-ca-cert\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.813579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.813548 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-envoy\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.813891 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.813872 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae9d702-f807-4040-b5e8-e140bead935f-istio-podinfo\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.872482 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.872420 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-istio-token\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.874165 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.874139 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llx47\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-kube-api-access-llx47\") pod \"e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:45.964663 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:45.964582 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:46.020114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:46.020081 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" event={"ID":"f946456d-2550-4e3f-adce-ff9fe63789cb","Type":"ContainerStarted","Data":"872b54f4dc7d7cb983e593fe64b36aabfc0a4fba18f9072e5f986324524a1597"} Oct 30 22:43:46.399023 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:46.398991 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l"] Oct 30 22:43:46.410358 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:43:46.407154 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae9d702_f807_4040_b5e8_e140bead935f.slice/crio-18c92902a021d04ba293e57672f96535b5161abea0c957b6b082bdd2c7431063 WatchSource:0}: Error finding container 18c92902a021d04ba293e57672f96535b5161abea0c957b6b082bdd2c7431063: Status 404 returned error can't find the container with id 18c92902a021d04ba293e57672f96535b5161abea0c957b6b082bdd2c7431063 Oct 30 22:43:46.412547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:46.412510 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:46.412638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:46.412595 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:46.412688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:46.412638 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:43:47.031206 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.031165 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" event={"ID":"8ae9d702-f807-4040-b5e8-e140bead935f","Type":"ContainerStarted","Data":"0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283"} Oct 30 22:43:47.031404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.031229 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" event={"ID":"8ae9d702-f807-4040-b5e8-e140bead935f","Type":"ContainerStarted","Data":"18c92902a021d04ba293e57672f96535b5161abea0c957b6b082bdd2c7431063"} Oct 30 22:43:47.032987 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.032961 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" event={"ID":"f946456d-2550-4e3f-adce-ff9fe63789cb","Type":"ContainerStarted","Data":"75851a0b1734799aa447db3bae09e3c76a2b54c0389194c2880b4963eacf1b4d"} Oct 30 22:43:47.033125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.033079 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:47.152347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.152243 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podStartSLOduration=2.152221517 podStartE2EDuration="2.152221517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:47.149780891 +0000 UTC m=+3447.449190772" watchObservedRunningTime="2026-10-30 22:43:47.152221517 +0000 UTC m=+3447.451631398" Oct 30 22:43:47.264870 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.264782 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" podStartSLOduration=2.330743517 podStartE2EDuration="3.264762301s" totalImagesPullingTime="934.018784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:47.261649156 +0000 UTC m=+3447.561059058" watchObservedRunningTime="2026-10-30 22:43:47.264762301 +0000 UTC m=+3447.564172182" Oct 30 22:43:47.912918 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:47.912882 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:43:47.964964 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.964933 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:47.970599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:47.970574 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:48.044483 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:48.044064 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:48.044848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:48.044829 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:43:48.854713 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:48.854676 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" podUID="07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" containerName="main" containerID="cri-o://b99a20acd8a72617e5cca47cd1ffbbb554cf8e378a03581f6d29ab51c8d5c6fd" gracePeriod=45 Oct 30 22:43:49.067228 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.067097 4991 generic.go:356] "Generic (PLEG): container finished" podID="07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" containerID="b99a20acd8a72617e5cca47cd1ffbbb554cf8e378a03581f6d29ab51c8d5c6fd" exitCode=0 Oct 30 22:43:49.069921 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.068408 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" event={"ID":"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977","Type":"ContainerDied","Data":"b99a20acd8a72617e5cca47cd1ffbbb554cf8e378a03581f6d29ab51c8d5c6fd"} Oct 30 22:43:49.131982 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.131958 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:43:49.160878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.160852 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-home\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-home\") pod \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " Oct 30 22:43:49.161005 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.160989 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-kube-api-access-t7fb6\" (UniqueName: \"kubernetes.io/projected/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-kube-api-access-t7fb6\") pod \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " Oct 30 22:43:49.161065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161021 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tmp-dir\") pod \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " Oct 30 22:43:49.161105 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161085 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tls-certs\" (UniqueName: \"kubernetes.io/secret/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tls-certs\") pod \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " Oct 30 22:43:49.161157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161141 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-dshm\") pod \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " Oct 30 22:43:49.161224 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161080 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-home" pod "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" (UID: "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.161296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161206 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-model-cache\") pod \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\" (UID: \"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977\") " Oct 30 22:43:49.161296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161262 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tmp-dir" pod "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" (UID: "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.161437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161413 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-model-cache" pod "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" (UID: "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.162114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161953 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.162114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161976 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.162114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.161989 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.163524 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.163499 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-kube-api-access-t7fb6" pod "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" (UID: "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977"). InnerVolumeSpecName "kube-api-access-t7fb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.163648 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.163605 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tls-certs" pod "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" (UID: "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:49.163722 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.163702 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-dshm" pod "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" (UID: "07c88d2f-f7e8-480a-a4ae-b7b3f7a83977"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.263373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.263344 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7fb6\" (UniqueName: \"kubernetes.io/projected/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-kube-api-access-t7fb6\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.263373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.263372 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.263534 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:49.263387 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:50.077581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:50.077552 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" Oct 30 22:43:50.077976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:50.077591 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl" event={"ID":"07c88d2f-f7e8-480a-a4ae-b7b3f7a83977","Type":"ContainerDied","Data":"ab187c7998ef0c81243c9227bb8b2e19dbf0812933b25db04f78f8da15856027"} Oct 30 22:43:50.077976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:50.077644 4991 scope.go:122] "RemoveContainer" containerID="b99a20acd8a72617e5cca47cd1ffbbb554cf8e378a03581f6d29ab51c8d5c6fd" Oct 30 22:43:50.214523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:50.214443 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz"] Oct 30 22:43:50.214661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:50.214620 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" containerID="cri-o://55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94" gracePeriod=30 Oct 30 22:43:50.252171 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:50.252138 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl"] Oct 30 22:43:50.255759 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:50.255735 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-435923/auth-test-model-435923-kserve-5dcd6cdb89-8kkhl"] Oct 30 22:43:51.170074 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:51.170042 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f"] Oct 30 22:43:51.799838 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:51.799803 4991 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:43:51.800028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:51.799855 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:43:51.875726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:51.875698 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" path="/var/lib/kubelet/pods/07c88d2f-f7e8-480a-a4ae-b7b3f7a83977/volumes" Oct 30 22:43:53.049922 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:53.049887 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:43:55.549002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.548965 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:43:55.660803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.660769 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-istio-token\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-istio-token\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.660803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.660814 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-credential-socket\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.661081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.660871 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-socket\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.661081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.660912 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-envoy\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.661081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.660946 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-certs\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.661081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.660980 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea52d3c2-640b-420d-a955-12544c223fbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea52d3c2-640b-420d-a955-12544c223fbc-istiod-ca-cert\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.661081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.661071 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-kube-api-access-rgn45\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-kube-api-access-rgn45\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.661387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.661102 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-data\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.661387 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.661140 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea52d3c2-640b-420d-a955-12544c223fbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea52d3c2-640b-420d-a955-12544c223fbc-istio-podinfo\") pod \"ea52d3c2-640b-420d-a955-12544c223fbc\" (UID: \"ea52d3c2-640b-420d-a955-12544c223fbc\") " Oct 30 22:43:55.662153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.662126 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea52d3c2-640b-420d-a955-12544c223fbc-istiod-ca-cert" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:55.662381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.662357 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-data" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:55.662499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.662476 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-certs" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:55.662659 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.662639 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-credential-socket" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:55.662803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.662778 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-socket" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:55.664782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.664755 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-envoy" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:55.664990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.664964 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-istio-token" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:55.665085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.664987 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-kube-api-access-rgn45" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "kube-api-access-rgn45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:55.665407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.665384 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea52d3c2-640b-420d-a955-12544c223fbc-istio-podinfo" pod "ea52d3c2-640b-420d-a955-12544c223fbc" (UID: "ea52d3c2-640b-420d-a955-12544c223fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:55.763986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.763949 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.763986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.763986 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.764199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.764001 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.764199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.764017 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.764199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.764031 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.764199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.764045 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea52d3c2-640b-420d-a955-12544c223fbc-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.764199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.764060 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgn45\" (UniqueName: \"kubernetes.io/projected/ea52d3c2-640b-420d-a955-12544c223fbc-kube-api-access-rgn45\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.764199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.764074 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea52d3c2-640b-420d-a955-12544c223fbc-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:55.764199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:55.764087 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea52d3c2-640b-420d-a955-12544c223fbc-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:56.122623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.122584 4991 generic.go:356] "Generic (PLEG): container finished" podID="ea52d3c2-640b-420d-a955-12544c223fbc" containerID="55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94" exitCode=0 Oct 30 22:43:56.122623 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.122632 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" Oct 30 22:43:56.122894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.122668 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" event={"ID":"ea52d3c2-640b-420d-a955-12544c223fbc","Type":"ContainerDied","Data":"55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94"} Oct 30 22:43:56.122894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.122737 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz" event={"ID":"ea52d3c2-640b-420d-a955-12544c223fbc","Type":"ContainerDied","Data":"6da43fc37d296e0369ca6cd3a868d84c2222a873cbaabe9bb33249b166633d3f"} Oct 30 22:43:56.122894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.122767 4991 scope.go:122] "RemoveContainer" containerID="55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94" Oct 30 22:43:56.133277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.133258 4991 scope.go:122] "RemoveContainer" containerID="55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94" Oct 30 22:43:56.133639 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:43:56.133611 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94\": container with ID starting with 55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94 not found: ID does not exist" containerID="55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94" Oct 30 22:43:56.133704 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.133650 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94"} err="failed to get container status \"55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94\": rpc error: code = NotFound desc = could not find container \"55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94\": container with ID starting with 55c8c3735e8a9508751cda9e3aa6b316b9aaf9143a95a9157e501e8e0b23bb94 not found: ID does not exist" Oct 30 22:43:56.287029 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.286998 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz"] Oct 30 22:43:56.333426 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:56.333400 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-hwdqz"] Oct 30 22:43:57.879457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:43:57.879423 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" path="/var/lib/kubelet/pods/ea52d3c2-640b-420d-a955-12544c223fbc/volumes" Oct 30 22:44:06.190993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.190907 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" podUID="2e88a477-7c03-4205-afe5-90418cfd3fd2" containerName="main" containerID="cri-o://422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23" gracePeriod=45 Oct 30 22:44:06.294795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.294760 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" podUID="2e88a477-7c03-4205-afe5-90418cfd3fd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.159:8000/ready\": dial tcp 10.132.0.159:8000: connect: connection refused" Oct 30 22:44:06.469271 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.469244 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:44:06.600416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600385 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-home\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-home\") pod \"2e88a477-7c03-4205-afe5-90418cfd3fd2\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " Oct 30 22:44:06.600592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600432 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e88a477-7c03-4205-afe5-90418cfd3fd2-kube-api-access-sfzwg\" (UniqueName: \"kubernetes.io/projected/2e88a477-7c03-4205-afe5-90418cfd3fd2-kube-api-access-sfzwg\") pod \"2e88a477-7c03-4205-afe5-90418cfd3fd2\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " Oct 30 22:44:06.600592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600480 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-model-cache\") pod \"2e88a477-7c03-4205-afe5-90418cfd3fd2\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " Oct 30 22:44:06.600592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600499 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-dshm\") pod \"2e88a477-7c03-4205-afe5-90418cfd3fd2\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " Oct 30 22:44:06.600779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600601 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-tmp-dir\") pod \"2e88a477-7c03-4205-afe5-90418cfd3fd2\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " Oct 30 22:44:06.600779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600640 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e88a477-7c03-4205-afe5-90418cfd3fd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e88a477-7c03-4205-afe5-90418cfd3fd2-tls-certs\") pod \"2e88a477-7c03-4205-afe5-90418cfd3fd2\" (UID: \"2e88a477-7c03-4205-afe5-90418cfd3fd2\") " Oct 30 22:44:06.600885 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600795 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-model-cache" pod "2e88a477-7c03-4205-afe5-90418cfd3fd2" (UID: "2e88a477-7c03-4205-afe5-90418cfd3fd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:06.600945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.600893 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-tmp-dir" pod "2e88a477-7c03-4205-afe5-90418cfd3fd2" (UID: "2e88a477-7c03-4205-afe5-90418cfd3fd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:06.601269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.601246 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:06.601452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.601274 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:06.601532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.601449 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-home" pod "2e88a477-7c03-4205-afe5-90418cfd3fd2" (UID: "2e88a477-7c03-4205-afe5-90418cfd3fd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:06.603154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.603127 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e88a477-7c03-4205-afe5-90418cfd3fd2-tls-certs" pod "2e88a477-7c03-4205-afe5-90418cfd3fd2" (UID: "2e88a477-7c03-4205-afe5-90418cfd3fd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:44:06.603252 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.603153 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e88a477-7c03-4205-afe5-90418cfd3fd2-kube-api-access-sfzwg" pod "2e88a477-7c03-4205-afe5-90418cfd3fd2" (UID: "2e88a477-7c03-4205-afe5-90418cfd3fd2"). InnerVolumeSpecName "kube-api-access-sfzwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:06.603514 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.603492 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-dshm" pod "2e88a477-7c03-4205-afe5-90418cfd3fd2" (UID: "2e88a477-7c03-4205-afe5-90418cfd3fd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:06.702522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.702465 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:06.702522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.702489 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzwg\" (UniqueName: \"kubernetes.io/projected/2e88a477-7c03-4205-afe5-90418cfd3fd2-kube-api-access-sfzwg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:06.702522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.702503 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e88a477-7c03-4205-afe5-90418cfd3fd2-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:06.702522 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:06.702512 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e88a477-7c03-4205-afe5-90418cfd3fd2-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:07.200577 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.200544 4991 generic.go:356] "Generic (PLEG): container finished" podID="2e88a477-7c03-4205-afe5-90418cfd3fd2" containerID="422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23" exitCode=0 Oct 30 22:44:07.200994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.200591 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" Oct 30 22:44:07.200994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.200625 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" event={"ID":"2e88a477-7c03-4205-afe5-90418cfd3fd2","Type":"ContainerDied","Data":"422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23"} Oct 30 22:44:07.200994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.200669 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f" event={"ID":"2e88a477-7c03-4205-afe5-90418cfd3fd2","Type":"ContainerDied","Data":"4030eec5518febb5f612adc611633e232f29f5da5e7b8a11b005e4eb6378c53b"} Oct 30 22:44:07.200994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.200686 4991 scope.go:122] "RemoveContainer" containerID="422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23" Oct 30 22:44:07.211850 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.211830 4991 scope.go:122] "RemoveContainer" containerID="422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23" Oct 30 22:44:07.212078 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:44:07.212060 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23\": container with ID starting with 422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23 not found: ID does not exist" containerID="422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23" Oct 30 22:44:07.212134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.212086 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23"} err="failed to get container status \"422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23\": rpc error: code = NotFound desc = could not find container \"422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23\": container with ID starting with 422e5d1816571ddd0f725d8617a1aeb363c16e57b6b18ee955463e7853405f23 not found: ID does not exist" Oct 30 22:44:07.359394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.359361 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f"] Oct 30 22:44:07.405576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.405549 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3b392d/auth-test-model-3b392d-kserve-69db4758bf-hx62f"] Oct 30 22:44:07.876060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:07.876025 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e88a477-7c03-4205-afe5-90418cfd3fd2" path="/var/lib/kubelet/pods/2e88a477-7c03-4205-afe5-90418cfd3fd2/volumes" Oct 30 22:44:17.910827 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:44:17.910791 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:44:23.124819 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.124777 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt"] Oct 30 22:44:23.125723 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125700 4991 cpu_manager.go:414] "containerMap: removing container" podUID="07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" containerName="main" Oct 30 22:44:23.125794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125726 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" containerName="main" Oct 30 22:44:23.125794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125750 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2e88a477-7c03-4205-afe5-90418cfd3fd2" containerName="main" Oct 30 22:44:23.125794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125758 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e88a477-7c03-4205-afe5-90418cfd3fd2" containerName="main" Oct 30 22:44:23.125794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125787 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" Oct 30 22:44:23.125924 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125796 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" Oct 30 22:44:23.125990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125977 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c88d2f-f7e8-480a-a4ae-b7b3f7a83977" containerName="main" Oct 30 22:44:23.126028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.125994 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e88a477-7c03-4205-afe5-90418cfd3fd2" containerName="main" Oct 30 22:44:23.126028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.126011 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea52d3c2-640b-420d-a955-12544c223fbc" containerName="istio-proxy" Oct 30 22:44:23.131701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.131676 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.260430 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.195659 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-dcde03-openshift-default-dockercfg-wchkc\"" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.277960 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-credential-socket\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278061 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-certs\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278101 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7rh\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-kube-api-access-kq7rh\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278137 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-token\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278242 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-socket\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278274 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istiod-ca-cert\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278305 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-data\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278357 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-podinfo\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.315665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.278380 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-envoy\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.340394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.335377 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt"] Oct 30 22:44:23.379745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.379655 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-socket\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.379745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.379716 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istiod-ca-cert\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.379990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.379753 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-data\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.379990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.379806 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-podinfo\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.379990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.379839 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-envoy\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.379990 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.379965 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-credential-socket\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.380214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.380057 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-certs\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.380214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.380096 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7rh\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-kube-api-access-kq7rh\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.380214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.380130 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-socket\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.380214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.380137 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-token\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.380601 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.380570 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-data\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.380676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.380602 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-certs\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.380757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.380734 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-credential-socket\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.381287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.381263 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istiod-ca-cert\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.383209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.383160 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-podinfo\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.383609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.383581 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-envoy\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.441063 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.441028 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7rh\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-kube-api-access-kq7rh\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.446633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.446583 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-token\") pod \"e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.479584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.479554 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:23.962073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.962040 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt"] Oct 30 22:44:23.969028 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:44:23.969000 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1440bd_b906_43d9_9a0b_9a2ed036abe7.slice/crio-8b03d94c70669614d1f53c42a809437b3fea49bccfa0937e5741573d08344b30 WatchSource:0}: Error finding container 8b03d94c70669614d1f53c42a809437b3fea49bccfa0937e5741573d08344b30: Status 404 returned error can't find the container with id 8b03d94c70669614d1f53c42a809437b3fea49bccfa0937e5741573d08344b30 Oct 30 22:44:23.972187 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.972153 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:44:23.972271 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.972222 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:44:23.972271 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:23.972249 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:44:24.375678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:24.375613 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" event={"ID":"df1440bd-b906-43d9-9a0b-9a2ed036abe7","Type":"ContainerStarted","Data":"40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677"} Oct 30 22:44:24.376201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:24.375723 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" event={"ID":"df1440bd-b906-43d9-9a0b-9a2ed036abe7","Type":"ContainerStarted","Data":"8b03d94c70669614d1f53c42a809437b3fea49bccfa0937e5741573d08344b30"} Oct 30 22:44:24.480241 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:24.480195 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:24.493346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:24.493274 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podStartSLOduration=1.49326079 podStartE2EDuration="1.49326079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:24.490084809 +0000 UTC m=+3484.789494713" watchObservedRunningTime="2026-10-30 22:44:24.49326079 +0000 UTC m=+3484.792670670" Oct 30 22:44:25.486484 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:25.486446 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:26.390905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.390820 4991 generic.go:356] "Generic (PLEG): container finished" podID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerID="15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8" exitCode=0 Oct 30 22:44:26.390905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.390895 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" event={"ID":"2ad2290c-3dae-4d9a-b985-8820a0611b5c","Type":"ContainerDied","Data":"15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8"} Oct 30 22:44:26.391301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.391279 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:26.391411 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.391343 4991 scope.go:122] "RemoveContainer" containerID="15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8" Oct 30 22:44:26.392439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.392421 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:44:26.392980 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.392955 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:44:26.393041 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.393021 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:44:26.393080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:26.393054 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:44:27.400467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:27.400430 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" event={"ID":"2ad2290c-3dae-4d9a-b985-8820a0611b5c","Type":"ContainerStarted","Data":"dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3"} Oct 30 22:44:28.101238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:28.101208 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:44:28.102462 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:28.102439 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:44:28.102537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:28.102484 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 22:44:28.251681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:28.251645 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:44:29.101933 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:29.101900 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:44:29.102453 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:29.102001 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 22:44:30.102266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:30.102230 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:44:30.103070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:30.102285 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 22:44:31.103724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:31.103683 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:44:31.104386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:31.104351 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 22:44:32.102075 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:32.102041 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:44:32.102287 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:32.102094 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 22:44:33.106265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:33.106235 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:44:33.107310 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:33.107290 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:44:38.184699 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.184668 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf"] Oct 30 22:44:38.187706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.187685 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.292598 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.292562 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6587ea\"/\"kube-root-ca.crt\"" Oct 30 22:44:38.292768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.292656 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6587ea\"/\"openshift-service-ca.crt\"" Oct 30 22:44:38.292768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.292703 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6587ea\"/\"default-dockercfg-g8rl2\"" Oct 30 22:44:38.296133 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.296114 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf"] Oct 30 22:44:38.328463 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.328436 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-home\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.328603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.328470 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8lc\" (UniqueName: \"kubernetes.io/projected/881b9739-bcdd-4865-bac5-144d3c7eb6af-kube-api-access-mh8lc\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.328603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.328581 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-dshm\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.328702 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.328623 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-tmp-dir\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.328779 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.328760 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881b9739-bcdd-4865-bac5-144d3c7eb6af-tls-certs\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.328816 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.328788 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-model-cache\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.347341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.345938 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6587ea\"/\"noten-model-6587ea-kserve-self-signed-certs\"" Oct 30 22:44:38.429761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.429725 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-dshm\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.429956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.429773 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-tmp-dir\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.429956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.429850 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881b9739-bcdd-4865-bac5-144d3c7eb6af-tls-certs\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.429956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.429869 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-model-cache\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.430149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.430014 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-home\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.430149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.430043 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8lc\" (UniqueName: \"kubernetes.io/projected/881b9739-bcdd-4865-bac5-144d3c7eb6af-kube-api-access-mh8lc\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.430381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.430310 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-model-cache\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.430537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.430378 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-home\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.430821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.430795 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-tmp-dir\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.432082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.432060 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-dshm\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.432729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.432711 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881b9739-bcdd-4865-bac5-144d3c7eb6af-tls-certs\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.488208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.488180 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8lc\" (UniqueName: \"kubernetes.io/projected/881b9739-bcdd-4865-bac5-144d3c7eb6af-kube-api-access-mh8lc\") pod \"noten-model-6587ea-kserve-5976bb4c8f-6rmlf\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.499104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.499080 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:38.880941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:38.880859 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf"] Oct 30 22:44:38.887114 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:44:38.887089 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881b9739_bcdd_4865_bac5_144d3c7eb6af.slice/crio-08d1a66df6651711981030aee709c4fac60ff3f8c75e0cdfe14ddc77805df658 WatchSource:0}: Error finding container 08d1a66df6651711981030aee709c4fac60ff3f8c75e0cdfe14ddc77805df658: Status 404 returned error can't find the container with id 08d1a66df6651711981030aee709c4fac60ff3f8c75e0cdfe14ddc77805df658 Oct 30 22:44:39.556483 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:39.556448 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" event={"ID":"881b9739-bcdd-4865-bac5-144d3c7eb6af","Type":"ContainerStarted","Data":"022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39"} Oct 30 22:44:39.556956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:39.556495 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" event={"ID":"881b9739-bcdd-4865-bac5-144d3c7eb6af","Type":"ContainerStarted","Data":"08d1a66df6651711981030aee709c4fac60ff3f8c75e0cdfe14ddc77805df658"} Oct 30 22:44:39.668918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:39.668695 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" podStartSLOduration=1.6686818479999999 podStartE2EDuration="1.668681848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:39.66836091 +0000 UTC m=+3499.967770792" watchObservedRunningTime="2026-10-30 22:44:39.668681848 +0000 UTC m=+3499.968091728" Oct 30 22:44:40.499664 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:40.499621 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:45.279381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.279344 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2"] Oct 30 22:44:45.282604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.282584 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.332209 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.332181 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9a60dd-serving-cert\"" Oct 30 22:44:45.384298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.384270 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2"] Oct 30 22:44:45.418837 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.418800 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm75v\" (UniqueName: \"kubernetes.io/projected/69284587-4eeb-462e-938c-e3b7f3bb85fb-kube-api-access-mm75v\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.418997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.418864 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69284587-4eeb-462e-938c-e3b7f3bb85fb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.419097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.419076 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-metrics-certs\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.419160 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.419113 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.520518 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.520479 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-metrics-certs\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.520823 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.520801 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.521006 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.520981 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm75v\" (UniqueName: \"kubernetes.io/projected/69284587-4eeb-462e-938c-e3b7f3bb85fb-kube-api-access-mm75v\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.521152 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.521051 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69284587-4eeb-462e-938c-e3b7f3bb85fb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.522041 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.522017 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69284587-4eeb-462e-938c-e3b7f3bb85fb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.522657 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:44:45.522641 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-serving-cert: secret "maas-api-e2e-extmod-a-9a60dd-serving-cert" not found Oct 30 22:44:45.523170 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:44:45.523156 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls podName:69284587-4eeb-462e-938c-e3b7f3bb85fb nodeName:}" failed. No retries permitted until 2026-10-30 22:44:46.023135393 +0000 UTC m=+3506.322545253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls") pod "maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" (UID: "69284587-4eeb-462e-938c-e3b7f3bb85fb") : secret "maas-api-e2e-extmod-a-9a60dd-serving-cert" not found Oct 30 22:44:45.550113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.549987 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-metrics-certs\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:45.582003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:45.581974 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm75v\" (UniqueName: \"kubernetes.io/projected/69284587-4eeb-462e-938c-e3b7f3bb85fb-kube-api-access-mm75v\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:46.028031 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:46.027992 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:46.030622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:46.030593 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:46.196161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:46.196126 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:46.588408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:46.588372 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2"] Oct 30 22:44:46.594309 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:44:46.594281 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69284587_4eeb_462e_938c_e3b7f3bb85fb.slice/crio-7911ae58a665a4a0edc1fd8c201c0df6044534913c7b4a47980f434618a5b548 WatchSource:0}: Error finding container 7911ae58a665a4a0edc1fd8c201c0df6044534913c7b4a47980f434618a5b548: Status 404 returned error can't find the container with id 7911ae58a665a4a0edc1fd8c201c0df6044534913c7b4a47980f434618a5b548 Oct 30 22:44:46.608959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:46.608929 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" event={"ID":"69284587-4eeb-462e-938c-e3b7f3bb85fb","Type":"ContainerStarted","Data":"7911ae58a665a4a0edc1fd8c201c0df6044534913c7b4a47980f434618a5b548"} Oct 30 22:44:47.617245 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:47.617209 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" event={"ID":"69284587-4eeb-462e-938c-e3b7f3bb85fb","Type":"ContainerStarted","Data":"2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b"} Oct 30 22:44:47.617589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:47.617408 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:47.735745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:47.735695 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" podStartSLOduration=1.837488472 podStartE2EDuration="2.735681456s" totalImagesPullingTime="898.192984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:47.733791665 +0000 UTC m=+3508.033201567" watchObservedRunningTime="2026-10-30 22:44:47.735681456 +0000 UTC m=+3508.035091336" Oct 30 22:44:47.914217 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:44:47.914122 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:44:48.500137 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:48.500104 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:48.512385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:48.512359 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:48.637530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:48.637502 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:44:48.761509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:48.761415 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz"] Oct 30 22:44:53.631300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:53.631270 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:44:55.729573 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:55.729541 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf"] Oct 30 22:44:55.912975 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:55.912940 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk"] Oct 30 22:44:55.913276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:55.913177 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" podUID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" containerName="istio-proxy" containerID="cri-o://323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab" gracePeriod=30 Oct 30 22:44:56.984789 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:56.984748 4991 patch_prober.go:29] interesting pod/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 22:44:56.985283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:44:56.984816 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" podUID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:45:00.229822 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.229780 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9"] Oct 30 22:45:00.234984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.234962 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.247866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.247843 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l"] Oct 30 22:45:00.252002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.251984 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.323672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.323641 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7ln\" (UniqueName: \"kubernetes.io/projected/25de8ea1-1777-45cf-a5dd-28601b772431-kube-api-access-gz7ln\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.323830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.323687 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30484c1-a249-46da-ab97-d367e38f2aa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.323830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.323713 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5prg\" (UniqueName: \"kubernetes.io/projected/e30484c1-a249-46da-ab97-d367e38f2aa1-kube-api-access-n5prg\") pod \"maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.323962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.323899 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25de8ea1-1777-45cf-a5dd-28601b772431-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.330526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.330499 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:45:00.383599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.383566 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l"] Oct 30 22:45:00.423193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.423158 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9"] Oct 30 22:45:00.425104 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.425074 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25de8ea1-1777-45cf-a5dd-28601b772431-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.425328 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.425295 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7ln\" (UniqueName: \"kubernetes.io/projected/25de8ea1-1777-45cf-a5dd-28601b772431-kube-api-access-gz7ln\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.425388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.425375 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30484c1-a249-46da-ab97-d367e38f2aa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.425429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.425413 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5prg\" (UniqueName: \"kubernetes.io/projected/e30484c1-a249-46da-ab97-d367e38f2aa1-kube-api-access-n5prg\") pod \"maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.425970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.425941 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25de8ea1-1777-45cf-a5dd-28601b772431-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.426408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.426384 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30484c1-a249-46da-ab97-d367e38f2aa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.483497 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.483438 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7ln\" (UniqueName: \"kubernetes.io/projected/25de8ea1-1777-45cf-a5dd-28601b772431-kube-api-access-gz7ln\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.483647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.483625 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5prg\" (UniqueName: \"kubernetes.io/projected/e30484c1-a249-46da-ab97-d367e38f2aa1-kube-api-access-n5prg\") pod \"maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.550806 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.550780 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:00.564547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.564525 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:00.936840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.936809 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9"] Oct 30 22:45:00.962552 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:00.962514 4991 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cd367f_a2ba_47a7_9a2f_2fd52b5b0405.slice/crio-323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:45:00.973347 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:45:00.973298 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25de8ea1_1777_45cf_a5dd_28601b772431.slice/crio-82283ef24efabdefeaea344ef59978da8940bbed05cbb09e6b566fddb62233fe WatchSource:0}: Error finding container 82283ef24efabdefeaea344ef59978da8940bbed05cbb09e6b566fddb62233fe: Status 404 returned error can't find the container with id 82283ef24efabdefeaea344ef59978da8940bbed05cbb09e6b566fddb62233fe Oct 30 22:45:00.987970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:00.987938 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l"] Oct 30 22:45:00.994076 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:45:00.994049 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30484c1_a249_46da_ab97_d367e38f2aa1.slice/crio-dfaf38eb541a2519fd5ef9474aae9afc0c37c2b341b2b1477d95d2e71e3dd6ff WatchSource:0}: Error finding container dfaf38eb541a2519fd5ef9474aae9afc0c37c2b341b2b1477d95d2e71e3dd6ff: Status 404 returned error can't find the container with id dfaf38eb541a2519fd5ef9474aae9afc0c37c2b341b2b1477d95d2e71e3dd6ff Oct 30 22:45:01.020105 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.020030 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:45:01.176414 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.176389 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:45:01.235395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235366 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istiod-ca-cert\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.235825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235422 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-certs\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.235825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235493 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-kube-api-access-9xmwz\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-kube-api-access-9xmwz\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.235825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235529 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-token\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.235825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235593 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-podinfo\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.235825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235665 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-data\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.235825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235737 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-credential-socket\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.236142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235860 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-socket\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.236142 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235911 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-envoy\") pod \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\" (UID: \"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405\") " Oct 30 22:45:01.236845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.235737 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istiod-ca-cert" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:01.236845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.236542 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-certs" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:01.236845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.236734 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-credential-socket" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:01.236845 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.236801 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-socket" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:01.237181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.237072 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-data" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:01.237622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.237598 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.237782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.237624 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.237782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.237639 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.237782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.237653 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.237782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.237667 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.238272 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.238244 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-token" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:01.238619 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.238597 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-podinfo" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:45:01.239090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.239068 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-kube-api-access-9xmwz" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "kube-api-access-9xmwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:01.239236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.239214 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-envoy" pod "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" (UID: "e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:01.338989 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.338956 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.339127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.338992 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xmwz\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-kube-api-access-9xmwz\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.339127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.339008 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.339127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.339022 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:01.726535 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.726489 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" event={"ID":"25de8ea1-1777-45cf-a5dd-28601b772431","Type":"ContainerStarted","Data":"82283ef24efabdefeaea344ef59978da8940bbed05cbb09e6b566fddb62233fe"} Oct 30 22:45:01.728064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.728038 4991 generic.go:356] "Generic (PLEG): container finished" podID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" containerID="323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab" exitCode=0 Oct 30 22:45:01.728204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.728105 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" Oct 30 22:45:01.728204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.728129 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" event={"ID":"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405","Type":"ContainerDied","Data":"323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab"} Oct 30 22:45:01.728204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.728181 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk" event={"ID":"e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405","Type":"ContainerDied","Data":"7ebd944c9d33f9a1f6c29ec7bc5a9f0c2f530d06bc51781bcad3de5975bf233b"} Oct 30 22:45:01.728400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.728211 4991 scope.go:122] "RemoveContainer" containerID="323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab" Oct 30 22:45:01.729731 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.729709 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" event={"ID":"e30484c1-a249-46da-ab97-d367e38f2aa1","Type":"ContainerStarted","Data":"dfaf38eb541a2519fd5ef9474aae9afc0c37c2b341b2b1477d95d2e71e3dd6ff"} Oct 30 22:45:01.739083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.739060 4991 scope.go:122] "RemoveContainer" containerID="323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab" Oct 30 22:45:01.739572 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:01.739394 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab\": container with ID starting with 323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab not found: ID does not exist" containerID="323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab" Oct 30 22:45:01.739572 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.739430 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab"} err="failed to get container status \"323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab\": rpc error: code = NotFound desc = could not find container \"323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab\": container with ID starting with 323dc1b83629b04511601f3937135cb906eac697ba479e7f3bf031f2606b94ab not found: ID does not exist" Oct 30 22:45:01.888942 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.887886 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk"] Oct 30 22:45:01.935726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:01.935685 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-6587ea-openshift-default-549c57899c-xv7qk"] Oct 30 22:45:03.788883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:03.786248 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" podUID="0d469a64-1f4f-43a0-b522-fe682affd4ba" containerName="main" containerID="cri-o://b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e" gracePeriod=45 Oct 30 22:45:03.879955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:03.879916 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" path="/var/lib/kubelet/pods/e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405/volumes" Oct 30 22:45:04.149243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.149208 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:45:04.190868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.190834 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-dshm\") pod \"0d469a64-1f4f-43a0-b522-fe682affd4ba\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " Oct 30 22:45:04.191040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.190886 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-home\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-home\") pod \"0d469a64-1f4f-43a0-b522-fe682affd4ba\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " Oct 30 22:45:04.191040 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.190938 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d469a64-1f4f-43a0-b522-fe682affd4ba-kube-api-access-h7b68\" (UniqueName: \"kubernetes.io/projected/0d469a64-1f4f-43a0-b522-fe682affd4ba-kube-api-access-h7b68\") pod \"0d469a64-1f4f-43a0-b522-fe682affd4ba\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " Oct 30 22:45:04.191184 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191113 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-tmp-dir\") pod \"0d469a64-1f4f-43a0-b522-fe682affd4ba\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " Oct 30 22:45:04.191250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191182 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-model-cache\") pod \"0d469a64-1f4f-43a0-b522-fe682affd4ba\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " Oct 30 22:45:04.191250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191225 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d469a64-1f4f-43a0-b522-fe682affd4ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d469a64-1f4f-43a0-b522-fe682affd4ba-tls-certs\") pod \"0d469a64-1f4f-43a0-b522-fe682affd4ba\" (UID: \"0d469a64-1f4f-43a0-b522-fe682affd4ba\") " Oct 30 22:45:04.191404 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191176 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-home" pod "0d469a64-1f4f-43a0-b522-fe682affd4ba" (UID: "0d469a64-1f4f-43a0-b522-fe682affd4ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:04.191455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191396 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-model-cache" pod "0d469a64-1f4f-43a0-b522-fe682affd4ba" (UID: "0d469a64-1f4f-43a0-b522-fe682affd4ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:04.191455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191409 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-tmp-dir" pod "0d469a64-1f4f-43a0-b522-fe682affd4ba" (UID: "0d469a64-1f4f-43a0-b522-fe682affd4ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:04.191773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191736 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:04.191773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191762 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:04.191930 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.191778 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:04.194080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.194048 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d469a64-1f4f-43a0-b522-fe682affd4ba-tls-certs" pod "0d469a64-1f4f-43a0-b522-fe682affd4ba" (UID: "0d469a64-1f4f-43a0-b522-fe682affd4ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:04.194162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.194111 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-dshm" pod "0d469a64-1f4f-43a0-b522-fe682affd4ba" (UID: "0d469a64-1f4f-43a0-b522-fe682affd4ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:04.194226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.194167 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d469a64-1f4f-43a0-b522-fe682affd4ba-kube-api-access-h7b68" pod "0d469a64-1f4f-43a0-b522-fe682affd4ba" (UID: "0d469a64-1f4f-43a0-b522-fe682affd4ba"). InnerVolumeSpecName "kube-api-access-h7b68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:04.293839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.293803 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d469a64-1f4f-43a0-b522-fe682affd4ba-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:04.294070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.294049 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7b68\" (UniqueName: \"kubernetes.io/projected/0d469a64-1f4f-43a0-b522-fe682affd4ba-kube-api-access-h7b68\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:04.294201 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.294186 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d469a64-1f4f-43a0-b522-fe682affd4ba-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:04.762972 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.762936 4991 generic.go:356] "Generic (PLEG): container finished" podID="0d469a64-1f4f-43a0-b522-fe682affd4ba" containerID="b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e" exitCode=0 Oct 30 22:45:04.763163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.762995 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" Oct 30 22:45:04.763163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.763012 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" event={"ID":"0d469a64-1f4f-43a0-b522-fe682affd4ba","Type":"ContainerDied","Data":"b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e"} Oct 30 22:45:04.763163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.763061 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz" event={"ID":"0d469a64-1f4f-43a0-b522-fe682affd4ba","Type":"ContainerDied","Data":"df641a1c343c7190a3709c917e082bfeecd60853930be2774cefc6e472ae6f27"} Oct 30 22:45:04.763163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.763075 4991 scope.go:122] "RemoveContainer" containerID="b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e" Oct 30 22:45:04.775587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.775548 4991 scope.go:122] "RemoveContainer" containerID="b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e" Oct 30 22:45:04.775947 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:04.775917 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e\": container with ID starting with b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e not found: ID does not exist" containerID="b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e" Oct 30 22:45:04.776047 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.775959 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e"} err="failed to get container status \"b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e\": rpc error: code = NotFound desc = could not find container \"b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e\": container with ID starting with b7c8aac5ed0b874dfb98e3cb1c51b54d9a76fe4ccc055652a9bee1efe07c1e6e not found: ID does not exist" Oct 30 22:45:04.928376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.928243 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz"] Oct 30 22:45:04.969732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:04.969690 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-74bb81/auth-test-model-74bb81-kserve-5d587bd5d5-8fxzz"] Oct 30 22:45:05.877111 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:05.877075 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d469a64-1f4f-43a0-b522-fe682affd4ba" path="/var/lib/kubelet/pods/0d469a64-1f4f-43a0-b522-fe682affd4ba/volumes" Oct 30 22:45:06.781669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:06.781585 4991 generic.go:356] "Generic (PLEG): container finished" podID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerID="8a24aa77b00a82089071daf33261d339a174f9053942b25af28a64a78fd44c9a" exitCode=60 Oct 30 22:45:06.782074 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:06.781664 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" event={"ID":"e30484c1-a249-46da-ab97-d367e38f2aa1","Type":"ContainerDied","Data":"8a24aa77b00a82089071daf33261d339a174f9053942b25af28a64a78fd44c9a"} Oct 30 22:45:06.782074 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:06.781877 4991 scope.go:122] "RemoveContainer" containerID="8a24aa77b00a82089071daf33261d339a174f9053942b25af28a64a78fd44c9a" Oct 30 22:45:06.783468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:06.783443 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" event={"ID":"25de8ea1-1777-45cf-a5dd-28601b772431","Type":"ContainerStarted","Data":"9525189900da8827d294e98a261da6a780b3460cf4ade52c288cf9f983dfe060"} Oct 30 22:45:07.003428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.003376 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" podStartSLOduration=2.07015035 podStartE2EDuration="7.003360234s" totalImagesPullingTime="4.933209884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:07.001991262 +0000 UTC m=+3527.301401162" watchObservedRunningTime="2026-10-30 22:45:07.003360234 +0000 UTC m=+3527.302770114" Oct 30 22:45:07.793427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.793395 4991 generic.go:356] "Generic (PLEG): container finished" podID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerID="2ef03388c0f3d54727ed62de84d87889befcb0d99cb159a02aaebc50f0d73f7f" exitCode=60 Oct 30 22:45:07.793875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.793482 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" event={"ID":"e30484c1-a249-46da-ab97-d367e38f2aa1","Type":"ContainerDied","Data":"2ef03388c0f3d54727ed62de84d87889befcb0d99cb159a02aaebc50f0d73f7f"} Oct 30 22:45:07.793875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.793534 4991 scope.go:122] "RemoveContainer" containerID="8a24aa77b00a82089071daf33261d339a174f9053942b25af28a64a78fd44c9a" Oct 30 22:45:07.793875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.793767 4991 scope.go:122] "RemoveContainer" containerID="2ef03388c0f3d54727ed62de84d87889befcb0d99cb159a02aaebc50f0d73f7f" Oct 30 22:45:07.794122 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:07.794099 4991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l_odh-ai-gateway-infra(e30484c1-a249-46da-ab97-d367e38f2aa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" Oct 30 22:45:07.795377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.795356 4991 generic.go:356] "Generic (PLEG): container finished" podID="25de8ea1-1777-45cf-a5dd-28601b772431" containerID="9525189900da8827d294e98a261da6a780b3460cf4ade52c288cf9f983dfe060" exitCode=60 Oct 30 22:45:07.795477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.795413 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" event={"ID":"25de8ea1-1777-45cf-a5dd-28601b772431","Type":"ContainerDied","Data":"9525189900da8827d294e98a261da6a780b3460cf4ade52c288cf9f983dfe060"} Oct 30 22:45:07.795633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:07.795619 4991 scope.go:122] "RemoveContainer" containerID="9525189900da8827d294e98a261da6a780b3460cf4ade52c288cf9f983dfe060" Oct 30 22:45:08.474061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.469955 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7"] Oct 30 22:45:08.474061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.470874 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" containerName="istio-proxy" Oct 30 22:45:08.474061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.470893 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" containerName="istio-proxy" Oct 30 22:45:08.474061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.470950 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0d469a64-1f4f-43a0-b522-fe682affd4ba" containerName="main" Oct 30 22:45:08.474061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.470959 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d469a64-1f4f-43a0-b522-fe682affd4ba" containerName="main" Oct 30 22:45:08.474061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.471107 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d469a64-1f4f-43a0-b522-fe682affd4ba" containerName="main" Oct 30 22:45:08.474061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.471127 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cd367f-a2ba-47a7-9a2f-2fd52b5b0405" containerName="istio-proxy" Oct 30 22:45:08.482480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.475251 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.524809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.524784 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1b5075-serving-cert\"" Oct 30 22:45:08.530050 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.530024 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7"] Oct 30 22:45:08.569423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.569391 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgdb\" (UniqueName: \"kubernetes.io/projected/cbacae8f-ac18-44ba-aedb-6ecb049209dc-kube-api-access-twgdb\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.569767 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.569742 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbacae8f-ac18-44ba-aedb-6ecb049209dc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.569868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.569793 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.569923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.569890 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-metrics-certs\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.671035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.670997 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgdb\" (UniqueName: \"kubernetes.io/projected/cbacae8f-ac18-44ba-aedb-6ecb049209dc-kube-api-access-twgdb\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.671205 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.671116 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbacae8f-ac18-44ba-aedb-6ecb049209dc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.671205 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.671167 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.671294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.671238 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-metrics-certs\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.671391 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:08.671368 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-serving-cert: secret "maas-api-e2e-extmod-b-1b5075-serving-cert" not found Oct 30 22:45:08.671464 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:08.671451 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls podName:cbacae8f-ac18-44ba-aedb-6ecb049209dc nodeName:}" failed. No retries permitted until 2026-10-30 22:45:09.171427546 +0000 UTC m=+3529.470837406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls") pod "maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" (UID: "cbacae8f-ac18-44ba-aedb-6ecb049209dc") : secret "maas-api-e2e-extmod-b-1b5075-serving-cert" not found Oct 30 22:45:08.671981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.671960 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbacae8f-ac18-44ba-aedb-6ecb049209dc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.673631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.673609 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-metrics-certs\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.742915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.742810 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgdb\" (UniqueName: \"kubernetes.io/projected/cbacae8f-ac18-44ba-aedb-6ecb049209dc-kube-api-access-twgdb\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:08.803985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.803958 4991 scope.go:122] "RemoveContainer" containerID="2ef03388c0f3d54727ed62de84d87889befcb0d99cb159a02aaebc50f0d73f7f" Oct 30 22:45:08.804462 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:08.804308 4991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l_odh-ai-gateway-infra(e30484c1-a249-46da-ab97-d367e38f2aa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" Oct 30 22:45:08.805671 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:08.805641 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" event={"ID":"25de8ea1-1777-45cf-a5dd-28601b772431","Type":"ContainerStarted","Data":"bc2ef66cb4da046fa56ce41f9fdd4e0b30c14e29b82eb8b2098ee9d3cf5a0960"} Oct 30 22:45:09.177474 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.177395 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:09.180090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.180066 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:09.405248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.405214 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:09.791949 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.791915 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7"] Oct 30 22:45:09.798744 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:45:09.798715 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbacae8f_ac18_44ba_aedb_6ecb049209dc.slice/crio-571eda469aba42e870777494e93752ecaef761b8b7f5952fd67ee62a72134e17 WatchSource:0}: Error finding container 571eda469aba42e870777494e93752ecaef761b8b7f5952fd67ee62a72134e17: Status 404 returned error can't find the container with id 571eda469aba42e870777494e93752ecaef761b8b7f5952fd67ee62a72134e17 Oct 30 22:45:09.815228 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.815200 4991 generic.go:356] "Generic (PLEG): container finished" podID="25de8ea1-1777-45cf-a5dd-28601b772431" containerID="bc2ef66cb4da046fa56ce41f9fdd4e0b30c14e29b82eb8b2098ee9d3cf5a0960" exitCode=60 Oct 30 22:45:09.815676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.815265 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" event={"ID":"25de8ea1-1777-45cf-a5dd-28601b772431","Type":"ContainerDied","Data":"bc2ef66cb4da046fa56ce41f9fdd4e0b30c14e29b82eb8b2098ee9d3cf5a0960"} Oct 30 22:45:09.815676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.815328 4991 scope.go:122] "RemoveContainer" containerID="9525189900da8827d294e98a261da6a780b3460cf4ade52c288cf9f983dfe060" Oct 30 22:45:09.815676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.815570 4991 scope.go:122] "RemoveContainer" containerID="bc2ef66cb4da046fa56ce41f9fdd4e0b30c14e29b82eb8b2098ee9d3cf5a0960" Oct 30 22:45:09.815937 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:09.815902 4991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9_odh-ai-gateway-infra(25de8ea1-1777-45cf-a5dd-28601b772431)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" Oct 30 22:45:09.816657 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:09.816634 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" event={"ID":"cbacae8f-ac18-44ba-aedb-6ecb049209dc","Type":"ContainerStarted","Data":"571eda469aba42e870777494e93752ecaef761b8b7f5952fd67ee62a72134e17"} Oct 30 22:45:10.750348 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:10.750284 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" podUID="881b9739-bcdd-4865-bac5-144d3c7eb6af" containerName="main" containerID="cri-o://022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39" gracePeriod=45 Oct 30 22:45:10.828156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:10.828130 4991 scope.go:122] "RemoveContainer" containerID="bc2ef66cb4da046fa56ce41f9fdd4e0b30c14e29b82eb8b2098ee9d3cf5a0960" Oct 30 22:45:10.828487 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:10.828461 4991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9_odh-ai-gateway-infra(25de8ea1-1777-45cf-a5dd-28601b772431)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" Oct 30 22:45:11.048007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.047972 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:45:11.115828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.115802 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881b9739-bcdd-4865-bac5-144d3c7eb6af-kube-api-access-mh8lc\" (UniqueName: \"kubernetes.io/projected/881b9739-bcdd-4865-bac5-144d3c7eb6af-kube-api-access-mh8lc\") pod \"881b9739-bcdd-4865-bac5-144d3c7eb6af\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " Oct 30 22:45:11.115959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.115852 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-home\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-home\") pod \"881b9739-bcdd-4865-bac5-144d3c7eb6af\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " Oct 30 22:45:11.115959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.115890 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-dshm\") pod \"881b9739-bcdd-4865-bac5-144d3c7eb6af\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " Oct 30 22:45:11.115959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.115948 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-model-cache\") pod \"881b9739-bcdd-4865-bac5-144d3c7eb6af\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " Oct 30 22:45:11.116147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.115991 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-tmp-dir\") pod \"881b9739-bcdd-4865-bac5-144d3c7eb6af\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " Oct 30 22:45:11.116147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.116031 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881b9739-bcdd-4865-bac5-144d3c7eb6af-tls-certs\" (UniqueName: \"kubernetes.io/secret/881b9739-bcdd-4865-bac5-144d3c7eb6af-tls-certs\") pod \"881b9739-bcdd-4865-bac5-144d3c7eb6af\" (UID: \"881b9739-bcdd-4865-bac5-144d3c7eb6af\") " Oct 30 22:45:11.116252 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.116138 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-home" pod "881b9739-bcdd-4865-bac5-144d3c7eb6af" (UID: "881b9739-bcdd-4865-bac5-144d3c7eb6af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:11.116305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.116241 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-tmp-dir" pod "881b9739-bcdd-4865-bac5-144d3c7eb6af" (UID: "881b9739-bcdd-4865-bac5-144d3c7eb6af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:11.116305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.116265 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-model-cache" pod "881b9739-bcdd-4865-bac5-144d3c7eb6af" (UID: "881b9739-bcdd-4865-bac5-144d3c7eb6af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:11.116608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.116578 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:11.116608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.116604 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:11.117025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.116619 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:11.118071 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.118048 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-dshm" pod "881b9739-bcdd-4865-bac5-144d3c7eb6af" (UID: "881b9739-bcdd-4865-bac5-144d3c7eb6af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:11.118155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.118108 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881b9739-bcdd-4865-bac5-144d3c7eb6af-tls-certs" pod "881b9739-bcdd-4865-bac5-144d3c7eb6af" (UID: "881b9739-bcdd-4865-bac5-144d3c7eb6af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:11.118155 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.118140 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881b9739-bcdd-4865-bac5-144d3c7eb6af-kube-api-access-mh8lc" pod "881b9739-bcdd-4865-bac5-144d3c7eb6af" (UID: "881b9739-bcdd-4865-bac5-144d3c7eb6af"). InnerVolumeSpecName "kube-api-access-mh8lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:11.217734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.217705 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8lc\" (UniqueName: \"kubernetes.io/projected/881b9739-bcdd-4865-bac5-144d3c7eb6af-kube-api-access-mh8lc\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:11.217734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.217732 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881b9739-bcdd-4865-bac5-144d3c7eb6af-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:11.217915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.217743 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881b9739-bcdd-4865-bac5-144d3c7eb6af-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:11.837229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.837191 4991 generic.go:356] "Generic (PLEG): container finished" podID="881b9739-bcdd-4865-bac5-144d3c7eb6af" containerID="022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39" exitCode=0 Oct 30 22:45:11.837820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.837233 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" Oct 30 22:45:11.837820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.837276 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" event={"ID":"881b9739-bcdd-4865-bac5-144d3c7eb6af","Type":"ContainerDied","Data":"022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39"} Oct 30 22:45:11.837820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.837350 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf" event={"ID":"881b9739-bcdd-4865-bac5-144d3c7eb6af","Type":"ContainerDied","Data":"08d1a66df6651711981030aee709c4fac60ff3f8c75e0cdfe14ddc77805df658"} Oct 30 22:45:11.837820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.837376 4991 scope.go:122] "RemoveContainer" containerID="022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39" Oct 30 22:45:11.839393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.839368 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" event={"ID":"cbacae8f-ac18-44ba-aedb-6ecb049209dc","Type":"ContainerStarted","Data":"3a8779b2b875412f56a7d0d20c5d85aaee64a16ac7e42ef9243c2ae312e447cb"} Oct 30 22:45:11.839473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.839420 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:11.847938 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.847922 4991 scope.go:122] "RemoveContainer" containerID="022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39" Oct 30 22:45:11.848204 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:11.848187 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39\": container with ID starting with 022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39 not found: ID does not exist" containerID="022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39" Oct 30 22:45:11.848273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.848212 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39"} err="failed to get container status \"022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39\": rpc error: code = NotFound desc = could not find container \"022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39\": container with ID starting with 022be1d148bd4ed048a69d09827581bf0228ef12b4a0b90abf6fe4f7afcf1a39 not found: ID does not exist" Oct 30 22:45:11.999388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:11.999343 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" podStartSLOduration=2.9833909050000003 podStartE2EDuration="3.999310246s" totalImagesPullingTime="1.015919341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:11.996921602 +0000 UTC m=+3532.296331502" watchObservedRunningTime="2026-10-30 22:45:11.999310246 +0000 UTC m=+3532.298720126" Oct 30 22:45:12.103609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:12.103538 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf"] Oct 30 22:45:12.106981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:12.106955 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6587ea/noten-model-6587ea-kserve-5976bb4c8f-6rmlf"] Oct 30 22:45:13.875124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:13.875093 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881b9739-bcdd-4865-bac5-144d3c7eb6af" path="/var/lib/kubelet/pods/881b9739-bcdd-4865-bac5-144d3c7eb6af/volumes" Oct 30 22:45:16.018343 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.018285 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x"] Oct 30 22:45:16.018897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.018883 4991 cpu_manager.go:414] "containerMap: removing container" podUID="881b9739-bcdd-4865-bac5-144d3c7eb6af" containerName="main" Oct 30 22:45:16.018968 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.018899 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881b9739-bcdd-4865-bac5-144d3c7eb6af" containerName="main" Oct 30 22:45:16.019008 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.019000 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="881b9739-bcdd-4865-bac5-144d3c7eb6af" containerName="main" Oct 30 22:45:16.034559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.034536 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.129507 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f4e241-openshift-default-dockercfg-x5j5c\"" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.129529 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x"] Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.200937 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-credential-socket\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201024 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7669c06-6d57-4547-b94b-9e5f07a55a36-istiod-ca-cert\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201057 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-certs\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201091 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-envoy\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201124 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-token\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201156 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-socket\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201360 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-data\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201436 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvb2l\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-kube-api-access-fvb2l\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.240971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.201649 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-podinfo\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.302813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302729 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-data\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.302813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302764 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvb2l\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-kube-api-access-fvb2l\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302819 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-podinfo\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302853 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-credential-socket\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302913 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7669c06-6d57-4547-b94b-9e5f07a55a36-istiod-ca-cert\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302937 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-certs\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302969 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-envoy\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.302999 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-token\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303246 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.303024 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-socket\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303246 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.303162 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-data\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.303397 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-credential-socket\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303609 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.303585 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-socket\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.303625 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-certs\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.303881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.303863 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7669c06-6d57-4547-b94b-9e5f07a55a36-istiod-ca-cert\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.305364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.305343 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-podinfo\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.305452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.305371 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-envoy\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.361726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.361657 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvb2l\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-kube-api-access-fvb2l\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.369121 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.369091 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-token\") pod \"e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:16.648508 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:16.648419 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:17.087562 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:17.087531 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x"] Oct 30 22:45:17.093581 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:45:17.093554 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7669c06_6d57_4547_b94b_9e5f07a55a36.slice/crio-f010dbbea166493042a602614379c10fbf9004fcd06c22f68759963ced4fc623 WatchSource:0}: Error finding container f010dbbea166493042a602614379c10fbf9004fcd06c22f68759963ced4fc623: Status 404 returned error can't find the container with id f010dbbea166493042a602614379c10fbf9004fcd06c22f68759963ced4fc623 Oct 30 22:45:17.096298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:17.096262 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:17.096423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:17.096346 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:17.096423 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:17.096381 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:17.852756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:17.852719 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:45:17.882131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:17.882096 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" event={"ID":"c7669c06-6d57-4547-b94b-9e5f07a55a36","Type":"ContainerStarted","Data":"e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302"} Oct 30 22:45:17.882288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:17.882147 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" event={"ID":"c7669c06-6d57-4547-b94b-9e5f07a55a36","Type":"ContainerStarted","Data":"f010dbbea166493042a602614379c10fbf9004fcd06c22f68759963ced4fc623"} Oct 30 22:45:17.911174 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:17.911143 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:45:18.091049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:18.090989 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podStartSLOduration=3.090973345 podStartE2EDuration="3.090973345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:18.081106954 +0000 UTC m=+3538.380516835" watchObservedRunningTime="2026-10-30 22:45:18.090973345 +0000 UTC m=+3538.390383224" Oct 30 22:45:18.649349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:18.649310 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:18.655064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:18.655040 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:18.888882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:18.888850 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:18.889736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:18.889713 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:45:21.867846 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:21.867815 4991 scope.go:122] "RemoveContainer" containerID="2ef03388c0f3d54727ed62de84d87889befcb0d99cb159a02aaebc50f0d73f7f" Oct 30 22:45:22.867934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:22.867908 4991 scope.go:122] "RemoveContainer" containerID="bc2ef66cb4da046fa56ce41f9fdd4e0b30c14e29b82eb8b2098ee9d3cf5a0960" Oct 30 22:45:22.919230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:22.919205 4991 generic.go:356] "Generic (PLEG): container finished" podID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerID="d450b24c91f129c73dfc224448be1f3a92576bd05f573b35a58e0a0b18218969" exitCode=60 Oct 30 22:45:22.919368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:22.919277 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" event={"ID":"e30484c1-a249-46da-ab97-d367e38f2aa1","Type":"ContainerDied","Data":"d450b24c91f129c73dfc224448be1f3a92576bd05f573b35a58e0a0b18218969"} Oct 30 22:45:22.919429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:22.919369 4991 scope.go:122] "RemoveContainer" containerID="2ef03388c0f3d54727ed62de84d87889befcb0d99cb159a02aaebc50f0d73f7f" Oct 30 22:45:22.919758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:22.919735 4991 scope.go:122] "RemoveContainer" containerID="d450b24c91f129c73dfc224448be1f3a92576bd05f573b35a58e0a0b18218969" Oct 30 22:45:22.920022 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:22.919999 4991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l_odh-ai-gateway-infra(e30484c1-a249-46da-ab97-d367e38f2aa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" Oct 30 22:45:23.930065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:23.929988 4991 generic.go:356] "Generic (PLEG): container finished" podID="25de8ea1-1777-45cf-a5dd-28601b772431" containerID="96c7ed0689c9824150b45f46d6283fdec0fb75967ffbc47e08170af8a0a807ae" exitCode=60 Oct 30 22:45:23.930065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:23.930050 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" event={"ID":"25de8ea1-1777-45cf-a5dd-28601b772431","Type":"ContainerDied","Data":"96c7ed0689c9824150b45f46d6283fdec0fb75967ffbc47e08170af8a0a807ae"} Oct 30 22:45:23.930633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:23.930104 4991 scope.go:122] "RemoveContainer" containerID="bc2ef66cb4da046fa56ce41f9fdd4e0b30c14e29b82eb8b2098ee9d3cf5a0960" Oct 30 22:45:23.930633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:23.930513 4991 scope.go:122] "RemoveContainer" containerID="96c7ed0689c9824150b45f46d6283fdec0fb75967ffbc47e08170af8a0a807ae" Oct 30 22:45:23.930815 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:23.930793 4991 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9_odh-ai-gateway-infra(25de8ea1-1777-45cf-a5dd-28601b772431)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" Oct 30 22:45:24.040946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.040912 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l"] Oct 30 22:45:24.202680 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.202659 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:24.316973 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.316943 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30484c1-a249-46da-ab97-d367e38f2aa1-kube-api-access-n5prg\" (UniqueName: \"kubernetes.io/projected/e30484c1-a249-46da-ab97-d367e38f2aa1-kube-api-access-n5prg\") pod \"e30484c1-a249-46da-ab97-d367e38f2aa1\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " Oct 30 22:45:24.317163 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.317070 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e30484c1-a249-46da-ab97-d367e38f2aa1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30484c1-a249-46da-ab97-d367e38f2aa1-maas-api-service-ca\") pod \"e30484c1-a249-46da-ab97-d367e38f2aa1\" (UID: \"e30484c1-a249-46da-ab97-d367e38f2aa1\") " Oct 30 22:45:24.317460 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.317436 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30484c1-a249-46da-ab97-d367e38f2aa1-maas-api-service-ca" pod "e30484c1-a249-46da-ab97-d367e38f2aa1" (UID: "e30484c1-a249-46da-ab97-d367e38f2aa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:24.318058 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.318021 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30484c1-a249-46da-ab97-d367e38f2aa1-maas-api-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:24.319294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.319265 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30484c1-a249-46da-ab97-d367e38f2aa1-kube-api-access-n5prg" pod "e30484c1-a249-46da-ab97-d367e38f2aa1" (UID: "e30484c1-a249-46da-ab97-d367e38f2aa1"). InnerVolumeSpecName "kube-api-access-n5prg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:24.419472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.419442 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5prg\" (UniqueName: \"kubernetes.io/projected/e30484c1-a249-46da-ab97-d367e38f2aa1-kube-api-access-n5prg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:24.940281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.940242 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" event={"ID":"e30484c1-a249-46da-ab97-d367e38f2aa1","Type":"ContainerDied","Data":"dfaf38eb541a2519fd5ef9474aae9afc0c37c2b341b2b1477d95d2e71e3dd6ff"} Oct 30 22:45:24.940747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.940304 4991 scope.go:122] "RemoveContainer" containerID="d450b24c91f129c73dfc224448be1f3a92576bd05f573b35a58e0a0b18218969" Oct 30 22:45:24.940747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:24.940264 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l" Oct 30 22:45:25.055926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.055893 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9"] Oct 30 22:45:25.185509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.185456 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l"] Oct 30 22:45:25.191352 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.188675 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b829bc-29890005-nwb6l"] Oct 30 22:45:25.335634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.335606 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:25.442046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.441984 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25de8ea1-1777-45cf-a5dd-28601b772431-kube-api-access-gz7ln\" (UniqueName: \"kubernetes.io/projected/25de8ea1-1777-45cf-a5dd-28601b772431-kube-api-access-gz7ln\") pod \"25de8ea1-1777-45cf-a5dd-28601b772431\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " Oct 30 22:45:25.442237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.442217 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25de8ea1-1777-45cf-a5dd-28601b772431-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25de8ea1-1777-45cf-a5dd-28601b772431-maas-api-service-ca\") pod \"25de8ea1-1777-45cf-a5dd-28601b772431\" (UID: \"25de8ea1-1777-45cf-a5dd-28601b772431\") " Oct 30 22:45:25.442532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.442505 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25de8ea1-1777-45cf-a5dd-28601b772431-maas-api-service-ca" pod "25de8ea1-1777-45cf-a5dd-28601b772431" (UID: "25de8ea1-1777-45cf-a5dd-28601b772431"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:25.442686 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.442671 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25de8ea1-1777-45cf-a5dd-28601b772431-maas-api-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:25.444070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.444045 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25de8ea1-1777-45cf-a5dd-28601b772431-kube-api-access-gz7ln" pod "25de8ea1-1777-45cf-a5dd-28601b772431" (UID: "25de8ea1-1777-45cf-a5dd-28601b772431"). InnerVolumeSpecName "kube-api-access-gz7ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:25.543516 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.543477 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7ln\" (UniqueName: \"kubernetes.io/projected/25de8ea1-1777-45cf-a5dd-28601b772431-kube-api-access-gz7ln\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:25.875358 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.875300 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" path="/var/lib/kubelet/pods/e30484c1-a249-46da-ab97-d367e38f2aa1/volumes" Oct 30 22:45:25.948239 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.948211 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" Oct 30 22:45:25.948602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.948240 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9" event={"ID":"25de8ea1-1777-45cf-a5dd-28601b772431","Type":"ContainerDied","Data":"82283ef24efabdefeaea344ef59978da8940bbed05cbb09e6b566fddb62233fe"} Oct 30 22:45:25.948602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:25.948299 4991 scope.go:122] "RemoveContainer" containerID="96c7ed0689c9824150b45f46d6283fdec0fb75967ffbc47e08170af8a0a807ae" Oct 30 22:45:26.105035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:26.105005 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9"] Oct 30 22:45:26.110771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:26.110749 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29890005-fpdx9"] Oct 30 22:45:27.876186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:27.876150 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" path="/var/lib/kubelet/pods/25de8ea1-1777-45cf-a5dd-28601b772431/volumes" Oct 30 22:45:41.576869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.576834 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg"] Oct 30 22:45:41.577498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577481 4991 cpu_manager.go:414] "containerMap: removing container" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577499 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577511 4991 cpu_manager.go:414] "containerMap: removing container" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577516 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577529 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577534 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577547 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577553 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577571 4991 cpu_manager.go:414] "containerMap: removing container" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577576 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577576 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577674 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577681 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.577906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577688 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:41.577906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577695 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:41.577906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577703 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:41.577906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.577708 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="25de8ea1-1777-45cf-a5dd-28601b772431" containerName="cleanup" Oct 30 22:45:41.585245 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.585224 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.641888 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.641854 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-031eee-openshift-default-dockercfg-b6z9j\"" Oct 30 22:45:41.694425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.694380 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg"] Oct 30 22:45:41.774212 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.774167 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrtq\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-kube-api-access-zdrtq\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.784506 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.784466 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-credential-socket\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.784696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.784595 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-podinfo\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.784696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.784632 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-socket\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.784696 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.784680 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-token\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.784805 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.784711 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-certs\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.784897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.784880 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-data\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.784985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.784973 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b242b11-a4fb-47d9-a92e-1233ae478508-istiod-ca-cert\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.785033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.785018 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-envoy\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.885879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.885800 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b242b11-a4fb-47d9-a92e-1233ae478508-istiod-ca-cert\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.885879 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.885848 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-envoy\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.885879 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrtq\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-kube-api-access-zdrtq\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.885904 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-credential-socket\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.885955 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-podinfo\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.885985 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-socket\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886016 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-token\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886041 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-certs\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886128 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-data\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886347 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-credential-socket\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886663 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886637 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b242b11-a4fb-47d9-a92e-1233ae478508-istiod-ca-cert\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886637 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-certs\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886704 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-data\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.886738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.886700 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-socket\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.888245 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.888223 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-envoy\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.888749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.888728 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-podinfo\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.947259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.947226 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-token\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:41.947468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:41.947414 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrtq\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-kube-api-access-zdrtq\") pod \"e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:42.198989 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:42.198899 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:42.671239 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:42.671202 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg"] Oct 30 22:45:42.676851 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:45:42.676817 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b242b11_a4fb_47d9_a92e_1233ae478508.slice/crio-cb1bda88778b95bbb468cfca23d8e0d0c8bdf753c911b6c1220e5104b4161396 WatchSource:0}: Error finding container cb1bda88778b95bbb468cfca23d8e0d0c8bdf753c911b6c1220e5104b4161396: Status 404 returned error can't find the container with id cb1bda88778b95bbb468cfca23d8e0d0c8bdf753c911b6c1220e5104b4161396 Oct 30 22:45:42.679596 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:42.679560 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:42.679695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:42.679634 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:42.679695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:42.679669 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:43.102708 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:43.102665 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" event={"ID":"3b242b11-a4fb-47d9-a92e-1233ae478508","Type":"ContainerStarted","Data":"f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25"} Oct 30 22:45:43.102903 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:43.102734 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" event={"ID":"3b242b11-a4fb-47d9-a92e-1233ae478508","Type":"ContainerStarted","Data":"cb1bda88778b95bbb468cfca23d8e0d0c8bdf753c911b6c1220e5104b4161396"} Oct 30 22:45:43.199512 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:43.199470 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:43.200856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:43.200829 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 22:45:43.201003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:43.200881 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 22:45:43.229767 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:43.229709 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podStartSLOduration=2.229691877 podStartE2EDuration="2.229691877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:43.21848605 +0000 UTC m=+3563.517895955" watchObservedRunningTime="2026-10-30 22:45:43.229691877 +0000 UTC m=+3563.529101758" Oct 30 22:45:44.204347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:44.204304 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:45.116781 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:45.116749 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:45.117704 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:45.117685 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:45:47.912483 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:45:47.912447 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:45:49.086471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:49.086428 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:45:49.086860 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:49.086506 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:45:50.182068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:50.182036 4991 generic.go:356] "Generic (PLEG): container finished" podID="8ae9d702-f807-4040-b5e8-e140bead935f" containerID="0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283" exitCode=0 Oct 30 22:45:50.182530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:50.182081 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" event={"ID":"8ae9d702-f807-4040-b5e8-e140bead935f","Type":"ContainerDied","Data":"0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283"} Oct 30 22:45:50.182610 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:50.182586 4991 scope.go:122] "RemoveContainer" containerID="0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283" Oct 30 22:45:50.184416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:50.184387 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:50.184507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:50.184457 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:50.184507 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:50.184495 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:45:51.356622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.356584 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc"] Oct 30 22:45:51.358305 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.358280 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:51.358520 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.358504 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30484c1-a249-46da-ab97-d367e38f2aa1" containerName="cleanup" Oct 30 22:45:51.370550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.370521 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.383868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.381878 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" event={"ID":"8ae9d702-f807-4040-b5e8-e140bead935f","Type":"ContainerStarted","Data":"26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e"} Oct 30 22:45:51.411214 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.411179 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc"] Oct 30 22:45:51.476791 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.474048 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-dcde03-serving-cert\"" Oct 30 22:45:51.511538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.511505 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-metrics-certs\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.511742 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.511687 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-maas-api-tls\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.511907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.511887 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9754p\" (UniqueName: \"kubernetes.io/projected/2d51cf74-f581-4f09-9d39-c5d421be1776-kube-api-access-9754p\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.511986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.511938 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d51cf74-f581-4f09-9d39-c5d421be1776-openshift-service-ca\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.612754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.612667 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9754p\" (UniqueName: \"kubernetes.io/projected/2d51cf74-f581-4f09-9d39-c5d421be1776-kube-api-access-9754p\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.612754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.612710 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d51cf74-f581-4f09-9d39-c5d421be1776-openshift-service-ca\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.612754 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.612759 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-metrics-certs\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.613474 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.612842 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-maas-api-tls\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.626366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.614303 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d51cf74-f581-4f09-9d39-c5d421be1776-openshift-service-ca\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.626366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.617760 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-maas-api-tls\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.670357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.665948 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-metrics-certs\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.708640 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.708610 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9754p\" (UniqueName: \"kubernetes.io/projected/2d51cf74-f581-4f09-9d39-c5d421be1776-kube-api-access-9754p\") pod \"maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:51.965688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.965585 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:45:51.965847 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.965817 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 22:45:51.965892 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.965855 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 22:45:51.996281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:51.996255 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:52.405421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:52.405387 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc"] Oct 30 22:45:52.411057 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:45:52.411031 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d51cf74_f581_4f09_9d39_c5d421be1776.slice/crio-a86329ced6b1fcc15e3356f7ebea9b86cb0b6b529a92e8623c0947ce1bf804b0 WatchSource:0}: Error finding container a86329ced6b1fcc15e3356f7ebea9b86cb0b6b529a92e8623c0947ce1bf804b0: Status 404 returned error can't find the container with id a86329ced6b1fcc15e3356f7ebea9b86cb0b6b529a92e8623c0947ce1bf804b0 Oct 30 22:45:52.968743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:52.965567 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 22:45:52.968743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:52.965627 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 22:45:53.398149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:53.398110 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" event={"ID":"2d51cf74-f581-4f09-9d39-c5d421be1776","Type":"ContainerStarted","Data":"a86329ced6b1fcc15e3356f7ebea9b86cb0b6b529a92e8623c0947ce1bf804b0"} Oct 30 22:45:53.964933 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:53.964897 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 22:45:53.965461 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:53.964950 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 22:45:54.406538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:54.406489 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" event={"ID":"2d51cf74-f581-4f09-9d39-c5d421be1776","Type":"ContainerStarted","Data":"a6f96cfbd2def13ef9f9603102f0cf33273284438fcb7c17c9da2ed5b59f9516"} Oct 30 22:45:54.406714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:54.406566 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:45:54.523341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:54.519445 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" podStartSLOduration=2.5586536239999997 podStartE2EDuration="3.519426958s" totalImagesPullingTime="960.773334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:54.517998353 +0000 UTC m=+3574.817408233" watchObservedRunningTime="2026-10-30 22:45:54.519426958 +0000 UTC m=+3574.818836840" Oct 30 22:45:54.965279 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:54.965242 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 22:45:54.965797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:54.965295 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 22:45:55.965187 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:55.965146 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 22:45:55.965421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:55.965212 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 22:45:56.969221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:56.969190 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:45:56.969715 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:56.969694 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:45:56.970536 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:45:56.970514 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:46:00.418475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:00.418436 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:46:17.911531 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:46:17.911495 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:46:20.522421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:20.522391 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:46:20.537251 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:20.537230 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:46:26.921961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:26.921923 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr"] Oct 30 22:46:26.926495 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:26.926477 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.032612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032580 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-credential-socket\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.032612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032616 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-envoy\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.032826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032729 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-certs\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.032826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032760 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-token\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.032923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032855 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032c362-3944-42cf-8576-2c74c8f9dca2-istiod-ca-cert\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.032923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032879 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-podinfo\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.033014 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032936 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shgw\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-kube-api-access-2shgw\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.033014 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.032961 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-data\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.033124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.033079 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-socket\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.035520 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.035498 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-293c80-openshift-default-dockercfg-v8q8t\"" Oct 30 22:46:27.133913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.133875 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-certs\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.133933 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-token\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134004 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032c362-3944-42cf-8576-2c74c8f9dca2-istiod-ca-cert\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134038 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-podinfo\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134079 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shgw\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-kube-api-access-2shgw\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134109 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-data\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134157 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-socket\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134385 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134361 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-certs\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134382 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-credential-socket\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134428 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-envoy\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.134912 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.134884 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032c362-3944-42cf-8576-2c74c8f9dca2-istiod-ca-cert\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.135061 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.135009 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-socket\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.135283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.135255 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-data\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.135283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.135275 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-credential-socket\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.137267 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.137247 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-podinfo\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.137417 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.137337 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-envoy\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.182067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.181999 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr"] Oct 30 22:46:27.194397 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.194363 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shgw\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-kube-api-access-2shgw\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.195233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.195205 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-token\") pod \"e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.243514 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.243431 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:27.433852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.433450 4991 patch_prober.go:29] interesting pod/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:46:27.433852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.433514 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:46:27.696981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.696898 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr"] Oct 30 22:46:27.703288 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:46:27.703259 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc032c362_3944_42cf_8576_2c74c8f9dca2.slice/crio-f00d31a1453f2ff4cc1e377a121cee3d56320e9c7db2b4a56e3b67ea700bac02 WatchSource:0}: Error finding container f00d31a1453f2ff4cc1e377a121cee3d56320e9c7db2b4a56e3b67ea700bac02: Status 404 returned error can't find the container with id f00d31a1453f2ff4cc1e377a121cee3d56320e9c7db2b4a56e3b67ea700bac02 Oct 30 22:46:27.706893 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.706856 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:46:27.707023 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.706944 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:46:27.707023 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:27.706993 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:46:28.707132 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:28.707083 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" event={"ID":"c032c362-3944-42cf-8576-2c74c8f9dca2","Type":"ContainerStarted","Data":"553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49"} Oct 30 22:46:28.707600 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:28.707161 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" event={"ID":"c032c362-3944-42cf-8576-2c74c8f9dca2","Type":"ContainerStarted","Data":"f00d31a1453f2ff4cc1e377a121cee3d56320e9c7db2b4a56e3b67ea700bac02"} Oct 30 22:46:28.873006 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:28.872946 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" podStartSLOduration=2.87292786 podStartE2EDuration="2.87292786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:28.868809998 +0000 UTC m=+3609.168219898" watchObservedRunningTime="2026-10-30 22:46:28.87292786 +0000 UTC m=+3609.172337743" Oct 30 22:46:29.243795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:29.243762 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:29.250391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:29.250365 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:29.716646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:29.716564 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:29.717722 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:29.717699 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:46:30.727210 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:30.727162 4991 generic.go:356] "Generic (PLEG): container finished" podID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerID="40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677" exitCode=0 Oct 30 22:46:30.727692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:30.727232 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" event={"ID":"df1440bd-b906-43d9-9a0b-9a2ed036abe7","Type":"ContainerDied","Data":"40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677"} Oct 30 22:46:30.727692 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:30.727623 4991 scope.go:122] "RemoveContainer" containerID="40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677" Oct 30 22:46:30.729396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:30.729367 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:46:30.729510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:30.729436 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:46:30.729510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:30.729465 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:46:31.751968 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:31.751931 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" event={"ID":"df1440bd-b906-43d9-9a0b-9a2ed036abe7","Type":"ContainerStarted","Data":"6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184"} Oct 30 22:46:32.479896 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:32.479860 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:46:32.480193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:32.480159 4991 patch_prober.go:29] interesting pod/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:46:32.480346 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:32.480220 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 22:46:33.480503 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:33.480470 4991 patch_prober.go:29] interesting pod/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:46:33.480992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:33.480525 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 22:46:34.480124 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:34.480092 4991 patch_prober.go:29] interesting pod/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:46:34.480379 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:34.480354 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 22:46:35.481077 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:35.481034 4991 patch_prober.go:29] interesting pod/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:46:35.481555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:35.481103 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 22:46:36.480734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:36.480696 4991 patch_prober.go:29] interesting pod/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:46:36.480919 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:36.480776 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 22:46:37.484972 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:37.484941 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:46:37.485478 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:37.485279 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:46:37.486048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:37.486026 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:46:47.912430 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:46:47.912387 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:46:52.452204 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.452171 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6"] Oct 30 22:46:52.469147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.469114 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.633308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.632006 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-f4e241-serving-cert\"" Oct 30 22:46:52.664005 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.662751 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6"] Oct 30 22:46:52.721967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.721838 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.721967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.721897 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-metrics-certs\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.721967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.721945 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.722298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.721990 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7t7\" (UniqueName: \"kubernetes.io/projected/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-kube-api-access-vl7t7\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.823371 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.823330 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.823545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.823379 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-metrics-certs\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.823545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.823425 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.823545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.823471 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7t7\" (UniqueName: \"kubernetes.io/projected/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-kube-api-access-vl7t7\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.826502 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.826478 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.829584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.829558 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.829584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.829579 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-metrics-certs\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.885110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.885059 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7t7\" (UniqueName: \"kubernetes.io/projected/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-kube-api-access-vl7t7\") pod \"maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:52.968807 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:52.968773 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:53.419262 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:53.419180 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6"] Oct 30 22:46:53.425183 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:46:53.425144 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfd066a_4b0b_4929_904f_aaf4d73d9dab.slice/crio-90583c416ed26a1ebede587fa522431d54f9ed11d3f9eba6a97e922d178abb8d WatchSource:0}: Error finding container 90583c416ed26a1ebede587fa522431d54f9ed11d3f9eba6a97e922d178abb8d: Status 404 returned error can't find the container with id 90583c416ed26a1ebede587fa522431d54f9ed11d3f9eba6a97e922d178abb8d Oct 30 22:46:53.925766 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:53.925728 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" event={"ID":"dbfd066a-4b0b-4929-904f-aaf4d73d9dab","Type":"ContainerStarted","Data":"90583c416ed26a1ebede587fa522431d54f9ed11d3f9eba6a97e922d178abb8d"} Oct 30 22:46:54.936430 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:54.936394 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" event={"ID":"dbfd066a-4b0b-4929-904f-aaf4d73d9dab","Type":"ContainerStarted","Data":"03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19"} Oct 30 22:46:54.936832 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:54.936495 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:46:55.048944 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:46:55.048897 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" podStartSLOduration=2.090707346 podStartE2EDuration="3.048884193s" totalImagesPullingTime="958.176847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:55.048648489 +0000 UTC m=+3635.348058370" watchObservedRunningTime="2026-10-30 22:46:55.048884193 +0000 UTC m=+3635.348294073" Oct 30 22:47:00.950003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:00.949910 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:47:17.914410 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:47:17.914364 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:47:26.214579 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:26.214547 4991 generic.go:356] "Generic (PLEG): container finished" podID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerID="e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302" exitCode=0 Oct 30 22:47:26.215036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:26.214620 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" event={"ID":"c7669c06-6d57-4547-b94b-9e5f07a55a36","Type":"ContainerDied","Data":"e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302"} Oct 30 22:47:26.215036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:26.214974 4991 scope.go:122] "RemoveContainer" containerID="e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302" Oct 30 22:47:26.216681 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:26.216654 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:47:26.216776 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:26.216725 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:47:26.216776 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:26.216757 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:47:27.253553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:27.253516 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" event={"ID":"c7669c06-6d57-4547-b94b-9e5f07a55a36","Type":"ContainerStarted","Data":"ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af"} Oct 30 22:47:27.650604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:27.650508 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:47:27.650839 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:27.650530 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 22:47:27.650996 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:27.650972 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 22:47:28.648925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:28.648894 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 22:47:28.649466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:28.648961 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 22:47:29.649815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:29.649757 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 22:47:29.650229 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:29.649834 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 22:47:30.651046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:30.651009 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 22:47:30.651676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:30.651642 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 22:47:31.650806 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:31.650764 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 22:47:31.651069 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:31.651039 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 22:47:32.653422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:32.653390 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:47:32.653877 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:32.653856 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:47:32.654588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:32.654566 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:47:47.912467 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:47:47.912429 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:47:53.017756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.017719 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh"] Oct 30 22:47:53.018419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.017928 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" containerID="cri-o://70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e" gracePeriod=30 Oct 30 22:47:53.508285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.508253 4991 generic.go:356] "Generic (PLEG): container finished" podID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerID="f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25" exitCode=0 Oct 30 22:47:53.508452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.508294 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" event={"ID":"3b242b11-a4fb-47d9-a92e-1233ae478508","Type":"ContainerDied","Data":"f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25"} Oct 30 22:47:53.508687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.508671 4991 scope.go:122] "RemoveContainer" containerID="f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25" Oct 30 22:47:53.510574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.510543 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:47:53.510667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.510613 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:47:53.510667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:53.510645 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:47:54.518708 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:54.518660 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" event={"ID":"3b242b11-a4fb-47d9-a92e-1233ae478508","Type":"ContainerStarted","Data":"79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065"} Oct 30 22:47:55.199685 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:55.199647 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:47:55.200803 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:55.200772 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 22:47:55.200934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:55.200834 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 22:47:56.199670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:56.199634 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 22:47:56.200183 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:56.199696 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 22:47:57.200211 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:57.200177 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 22:47:57.200636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:57.200227 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 22:47:58.200416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.200023 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 22:47:58.200416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.200091 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 22:47:58.387741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.387711 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:47:58.497550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497521 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-certs\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.497729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497634 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-credential-socket\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.497729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497691 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istiod-ca-cert\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.497822 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497751 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-token\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-token\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.497865 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497816 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-socket\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.497865 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497829 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-certs" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:58.497865 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497848 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-data\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.497997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497891 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-envoy\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.497997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497896 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-credential-socket" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:58.497997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.497966 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-podinfo\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.498173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498036 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-kube-api-access-5t2gg\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-kube-api-access-5t2gg\") pod \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\" (UID: \"26e6ac07-556a-4cf1-a45d-4a6a02c9805b\") " Oct 30 22:47:58.498173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498039 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-socket" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:58.498173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498077 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-data" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:58.498173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498131 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istiod-ca-cert" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:58.498498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498472 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.498498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498500 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.498701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498510 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.498701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498518 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.498701 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.498526 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.500304 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.500279 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-podinfo" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:58.500422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.500343 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-envoy" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:58.500422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.500379 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-kube-api-access-5t2gg" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "kube-api-access-5t2gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:58.500422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.500393 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-token" pod "26e6ac07-556a-4cf1-a45d-4a6a02c9805b" (UID: "26e6ac07-556a-4cf1-a45d-4a6a02c9805b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:58.574764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.574691 4991 generic.go:356] "Generic (PLEG): container finished" podID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerID="70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e" exitCode=0 Oct 30 22:47:58.574918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.574900 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" event={"ID":"26e6ac07-556a-4cf1-a45d-4a6a02c9805b","Type":"ContainerDied","Data":"70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e"} Oct 30 22:47:58.574962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.574944 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" event={"ID":"26e6ac07-556a-4cf1-a45d-4a6a02c9805b","Type":"ContainerDied","Data":"e2dcbf22a09f8d788a58ba820d9cd9160436f0522f0189b22f7b081fabb3099a"} Oct 30 22:47:58.574999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.574966 4991 scope.go:122] "RemoveContainer" containerID="70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e" Oct 30 22:47:58.575109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.575098 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh" Oct 30 22:47:58.591288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.591271 4991 scope.go:122] "RemoveContainer" containerID="70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e" Oct 30 22:47:58.591645 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:47:58.591618 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e\": container with ID starting with 70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e not found: ID does not exist" containerID="70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e" Oct 30 22:47:58.591744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.591655 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e"} err="failed to get container status \"70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e\": rpc error: code = NotFound desc = could not find container \"70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e\": container with ID starting with 70d8cc8ad53ff2a05b5596990ebadacaf58f46abde221275cb005855d380700e not found: ID does not exist" Oct 30 22:47:58.599806 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.599777 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.599925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.599811 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2gg\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-kube-api-access-5t2gg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.599925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.599827 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.599925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.599844 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6ac07-556a-4cf1-a45d-4a6a02c9805b-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.763931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.763897 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh"] Oct 30 22:47:58.774450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:58.774416 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-d8xwh"] Oct 30 22:47:59.199751 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:59.199721 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 22:47:59.199939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:59.199773 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 22:47:59.885205 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:47:59.885162 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" path="/var/lib/kubelet/pods/26e6ac07-556a-4cf1-a45d-4a6a02c9805b/volumes" Oct 30 22:48:00.117308 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:00.117275 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:48:00.204620 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:00.204552 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:48:00.591532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:00.591498 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:48:11.293391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.293355 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv"] Oct 30 22:48:11.293945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.293930 4991 cpu_manager.go:414] "containerMap: removing container" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" Oct 30 22:48:11.293992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.293945 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" Oct 30 22:48:11.294078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.294068 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e6ac07-556a-4cf1-a45d-4a6a02c9805b" containerName="istio-proxy" Oct 30 22:48:11.299265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.299244 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.393126 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.393089 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-293c80-serving-cert\"" Oct 30 22:48:11.400895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.400870 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv"] Oct 30 22:48:11.468050 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.468014 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-maas-api-tls\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.468221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.468147 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f395d08-cdec-44fe-84ba-56084638c996-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.468221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.468175 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-metrics-certs\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.468420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.468243 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmsbr\" (UniqueName: \"kubernetes.io/projected/0f395d08-cdec-44fe-84ba-56084638c996-kube-api-access-cmsbr\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.570162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.570073 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-maas-api-tls\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.571059 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.571012 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f395d08-cdec-44fe-84ba-56084638c996-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.571059 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.571054 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-metrics-certs\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.571265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.571083 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmsbr\" (UniqueName: \"kubernetes.io/projected/0f395d08-cdec-44fe-84ba-56084638c996-kube-api-access-cmsbr\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.572279 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.572252 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f395d08-cdec-44fe-84ba-56084638c996-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.573499 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.573471 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-maas-api-tls\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.575129 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.575109 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-metrics-certs\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.629091 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.629068 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmsbr\" (UniqueName: \"kubernetes.io/projected/0f395d08-cdec-44fe-84ba-56084638c996-kube-api-access-cmsbr\") pod \"maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:11.913555 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:11.913473 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:12.302876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:12.302844 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv"] Oct 30 22:48:12.310296 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:48:12.310264 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f395d08_cdec_44fe_84ba_56084638c996.slice/crio-7519e0c916a85f08cc954c396f8456921c4e0616f9d32d8b5475b9c75742b96c WatchSource:0}: Error finding container 7519e0c916a85f08cc954c396f8456921c4e0616f9d32d8b5475b9c75742b96c: Status 404 returned error can't find the container with id 7519e0c916a85f08cc954c396f8456921c4e0616f9d32d8b5475b9c75742b96c Oct 30 22:48:12.680797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:12.680716 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" event={"ID":"0f395d08-cdec-44fe-84ba-56084638c996","Type":"ContainerStarted","Data":"7519e0c916a85f08cc954c396f8456921c4e0616f9d32d8b5475b9c75742b96c"} Oct 30 22:48:13.689144 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:13.689105 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" event={"ID":"0f395d08-cdec-44fe-84ba-56084638c996","Type":"ContainerStarted","Data":"4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883"} Oct 30 22:48:13.689554 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:13.689171 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:13.814881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:13.805413 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" podStartSLOduration=1.897556268 podStartE2EDuration="2.805393643s" totalImagesPullingTime="907.837375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:13.803748359 +0000 UTC m=+3714.103158241" watchObservedRunningTime="2026-10-30 22:48:13.805393643 +0000 UTC m=+3714.104803525" Oct 30 22:48:17.912081 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:48:17.912041 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:48:19.701672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:19.701633 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:48:47.912518 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:48:47.912478 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:48:49.961972 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:49.961934 4991 generic.go:356] "Generic (PLEG): container finished" podID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerID="553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49" exitCode=0 Oct 30 22:48:49.962549 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:49.962007 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" event={"ID":"c032c362-3944-42cf-8576-2c74c8f9dca2","Type":"ContainerDied","Data":"553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49"} Oct 30 22:48:49.962549 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:49.962432 4991 scope.go:122] "RemoveContainer" containerID="553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49" Oct 30 22:48:49.964064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:49.964035 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:48:49.964157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:49.964103 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:48:49.964157 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:49.964134 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:48:50.974582 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:50.974545 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" event={"ID":"c032c362-3944-42cf-8576-2c74c8f9dca2","Type":"ContainerStarted","Data":"ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80"} Oct 30 22:48:51.248412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:51.246790 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 22:48:51.248412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:51.246847 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 22:48:51.248412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:51.246491 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:48:52.249354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:52.244793 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 22:48:52.249354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:52.244856 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 22:48:53.245608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:53.245571 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 22:48:53.245878 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:53.245853 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 22:48:54.247344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:54.246971 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 22:48:54.247344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:54.247041 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 22:48:55.245663 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:55.245628 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 22:48:55.245960 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:55.245931 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 22:48:56.248425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:56.248383 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:48:56.248818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:56.248752 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:48:56.249391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:48:56.249370 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:49:08.618687 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:08.618651 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw"] Oct 30 22:49:08.619097 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:08.618851 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" containerID="cri-o://254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085" gracePeriod=30 Oct 30 22:49:08.926221 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:08.926137 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc"] Oct 30 22:49:08.926448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:08.926397 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" podUID="2d51cf74-f581-4f09-9d39-c5d421be1776" containerName="maas-api" containerID="cri-o://a6f96cfbd2def13ef9f9603102f0cf33273284438fcb7c17c9da2ed5b59f9516" gracePeriod=30 Oct 30 22:49:09.104994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.104952 4991 generic.go:356] "Generic (PLEG): container finished" podID="2d51cf74-f581-4f09-9d39-c5d421be1776" containerID="a6f96cfbd2def13ef9f9603102f0cf33273284438fcb7c17c9da2ed5b59f9516" exitCode=0 Oct 30 22:49:09.105177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.105035 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" event={"ID":"2d51cf74-f581-4f09-9d39-c5d421be1776","Type":"ContainerDied","Data":"a6f96cfbd2def13ef9f9603102f0cf33273284438fcb7c17c9da2ed5b59f9516"} Oct 30 22:49:09.270066 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.270039 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:49:09.399958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.399918 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-metrics-certs\") pod \"2d51cf74-f581-4f09-9d39-c5d421be1776\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " Oct 30 22:49:09.400152 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.400064 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-maas-api-tls\") pod \"2d51cf74-f581-4f09-9d39-c5d421be1776\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " Oct 30 22:49:09.400152 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.400114 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d51cf74-f581-4f09-9d39-c5d421be1776-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d51cf74-f581-4f09-9d39-c5d421be1776-openshift-service-ca\") pod \"2d51cf74-f581-4f09-9d39-c5d421be1776\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " Oct 30 22:49:09.400295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.400162 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d51cf74-f581-4f09-9d39-c5d421be1776-kube-api-access-9754p\" (UniqueName: \"kubernetes.io/projected/2d51cf74-f581-4f09-9d39-c5d421be1776-kube-api-access-9754p\") pod \"2d51cf74-f581-4f09-9d39-c5d421be1776\" (UID: \"2d51cf74-f581-4f09-9d39-c5d421be1776\") " Oct 30 22:49:09.400734 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.400703 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d51cf74-f581-4f09-9d39-c5d421be1776-openshift-service-ca" pod "2d51cf74-f581-4f09-9d39-c5d421be1776" (UID: "2d51cf74-f581-4f09-9d39-c5d421be1776"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:09.400916 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.400830 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d51cf74-f581-4f09-9d39-c5d421be1776-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:09.403032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.403001 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-metrics-certs" pod "2d51cf74-f581-4f09-9d39-c5d421be1776" (UID: "2d51cf74-f581-4f09-9d39-c5d421be1776"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:09.403381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.403352 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-maas-api-tls" pod "2d51cf74-f581-4f09-9d39-c5d421be1776" (UID: "2d51cf74-f581-4f09-9d39-c5d421be1776"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:09.403477 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.403373 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d51cf74-f581-4f09-9d39-c5d421be1776-kube-api-access-9754p" pod "2d51cf74-f581-4f09-9d39-c5d421be1776" (UID: "2d51cf74-f581-4f09-9d39-c5d421be1776"). InnerVolumeSpecName "kube-api-access-9754p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:09.502646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.502585 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:09.502646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.502610 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d51cf74-f581-4f09-9d39-c5d421be1776-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:09.502646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:09.502620 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9754p\" (UniqueName: \"kubernetes.io/projected/2d51cf74-f581-4f09-9d39-c5d421be1776-kube-api-access-9754p\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:10.114048 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:10.114015 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" event={"ID":"2d51cf74-f581-4f09-9d39-c5d421be1776","Type":"ContainerDied","Data":"a86329ced6b1fcc15e3356f7ebea9b86cb0b6b529a92e8623c0947ce1bf804b0"} Oct 30 22:49:10.114561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:10.114059 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc" Oct 30 22:49:10.114561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:10.114075 4991 scope.go:122] "RemoveContainer" containerID="a6f96cfbd2def13ef9f9603102f0cf33273284438fcb7c17c9da2ed5b59f9516" Oct 30 22:49:10.274781 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:10.274702 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc"] Oct 30 22:49:10.319594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:10.319566 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dcde03-568b54cdc9-fpqsc"] Oct 30 22:49:11.876313 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:11.876279 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d51cf74-f581-4f09-9d39-c5d421be1776" path="/var/lib/kubelet/pods/2d51cf74-f581-4f09-9d39-c5d421be1776/volumes" Oct 30 22:49:12.910519 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:12.910486 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg"] Oct 30 22:49:12.910943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:12.910715 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" containerID="cri-o://79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065" gracePeriod=30 Oct 30 22:49:13.171767 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.171380 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:49:13.171767 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.171444 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:49:13.884002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.883947 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:49:13.976919 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.976887 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-credential-socket\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.976973 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-data\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977028 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-istio-token\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-istio-token\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977065 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768d126b-8220-489e-91df-595b1bdce9b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d126b-8220-489e-91df-595b1bdce9b6-istiod-ca-cert\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977133 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-envoy\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977148 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-credential-socket" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977229 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-data" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977248 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-kube-api-access-s5fdj\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-kube-api-access-s5fdj\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977303 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-socket\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977370 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-certs\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977408 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/768d126b-8220-489e-91df-595b1bdce9b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d126b-8220-489e-91df-595b1bdce9b6-istio-podinfo\") pod \"768d126b-8220-489e-91df-595b1bdce9b6\" (UID: \"768d126b-8220-489e-91df-595b1bdce9b6\") " Oct 30 22:49:13.977876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977505 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768d126b-8220-489e-91df-595b1bdce9b6-istiod-ca-cert" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:13.977876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977586 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-socket" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:13.977876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.977705 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-certs" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:13.978128 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.978107 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d126b-8220-489e-91df-595b1bdce9b6-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.978189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.978131 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.978189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.978145 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.978189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.978159 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.978189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.978173 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.979625 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.979588 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-envoy" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:13.979759 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.979708 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-istio-token" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:13.979826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.979806 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-kube-api-access-s5fdj" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "kube-api-access-s5fdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:13.980000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:13.979975 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/768d126b-8220-489e-91df-595b1bdce9b6-istio-podinfo" pod "768d126b-8220-489e-91df-595b1bdce9b6" (UID: "768d126b-8220-489e-91df-595b1bdce9b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:49:14.080012 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.079980 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.080012 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.080009 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d126b-8220-489e-91df-595b1bdce9b6-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.080194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.080029 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5fdj\" (UniqueName: \"kubernetes.io/projected/768d126b-8220-489e-91df-595b1bdce9b6-kube-api-access-s5fdj\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.080194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.080043 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d126b-8220-489e-91df-595b1bdce9b6-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.176950 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.176871 4991 generic.go:356] "Generic (PLEG): container finished" podID="768d126b-8220-489e-91df-595b1bdce9b6" containerID="254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085" exitCode=0 Oct 30 22:49:14.176950 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.176924 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" Oct 30 22:49:14.177127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.176961 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" event={"ID":"768d126b-8220-489e-91df-595b1bdce9b6","Type":"ContainerDied","Data":"254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085"} Oct 30 22:49:14.177127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.177024 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw" event={"ID":"768d126b-8220-489e-91df-595b1bdce9b6","Type":"ContainerDied","Data":"ba95dd2a2f99d06c484daaab5c2564735388b4fd85b204f34baa107ad4e89067"} Oct 30 22:49:14.177127 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.177056 4991 scope.go:122] "RemoveContainer" containerID="254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085" Oct 30 22:49:14.188156 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.188140 4991 scope.go:122] "RemoveContainer" containerID="7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd" Oct 30 22:49:14.198790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.198770 4991 scope.go:122] "RemoveContainer" containerID="254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085" Oct 30 22:49:14.199022 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:14.199004 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085\": container with ID starting with 254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085 not found: ID does not exist" containerID="254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085" Oct 30 22:49:14.199071 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.199044 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085"} err="failed to get container status \"254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085\": rpc error: code = NotFound desc = could not find container \"254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085\": container with ID starting with 254f8394dc1602c3dcf9e3aacd2a4cad126f2df6d8eda58fb75d4a7ad6a6b085 not found: ID does not exist" Oct 30 22:49:14.199112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.199074 4991 scope.go:122] "RemoveContainer" containerID="7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd" Oct 30 22:49:14.199315 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:14.199296 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd\": container with ID starting with 7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd not found: ID does not exist" containerID="7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd" Oct 30 22:49:14.199398 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.199348 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd"} err="failed to get container status \"7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd\": rpc error: code = NotFound desc = could not find container \"7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd\": container with ID starting with 7c759e4c0d4355785aa0442b96104df2d93b09082c4059331fced5cb25eec9cd not found: ID does not exist" Oct 30 22:49:14.347537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.347504 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw"] Oct 30 22:49:14.353616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:14.353586 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-rkcbw"] Oct 30 22:49:15.592786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:15.592752 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:49:15.593286 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:15.592804 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:49:15.878667 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:15.878576 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768d126b-8220-489e-91df-595b1bdce9b6" path="/var/lib/kubelet/pods/768d126b-8220-489e-91df-595b1bdce9b6/volumes" Oct 30 22:49:17.912366 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:17.912301 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:49:18.193509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.193487 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:49:18.218113 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.218086 4991 generic.go:356] "Generic (PLEG): container finished" podID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerID="79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065" exitCode=0 Oct 30 22:49:18.218247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.218135 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" Oct 30 22:49:18.218247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.218181 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" event={"ID":"3b242b11-a4fb-47d9-a92e-1233ae478508","Type":"ContainerDied","Data":"79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065"} Oct 30 22:49:18.218407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.218249 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg" event={"ID":"3b242b11-a4fb-47d9-a92e-1233ae478508","Type":"ContainerDied","Data":"cb1bda88778b95bbb468cfca23d8e0d0c8bdf753c911b6c1220e5104b4161396"} Oct 30 22:49:18.218407 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.218262 4991 scope.go:122] "RemoveContainer" containerID="79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065" Oct 30 22:49:18.230149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230121 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-credential-socket\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230206 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-podinfo\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230256 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-socket\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230330 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-kube-api-access-zdrtq\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-kube-api-access-zdrtq\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230359 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-envoy\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230418 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-token\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-token\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230471 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-certs\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230558 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-data\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.230719 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230653 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b242b11-a4fb-47d9-a92e-1233ae478508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b242b11-a4fb-47d9-a92e-1233ae478508-istiod-ca-cert\") pod \"3b242b11-a4fb-47d9-a92e-1233ae478508\" (UID: \"3b242b11-a4fb-47d9-a92e-1233ae478508\") " Oct 30 22:49:18.231115 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.230938 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-socket" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:18.231289 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.231266 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-credential-socket" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:18.231633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.231600 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-certs" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:18.231852 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.231835 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-data" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:18.232172 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.232150 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b242b11-a4fb-47d9-a92e-1233ae478508-istiod-ca-cert" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:18.232368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.232311 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.233602 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.233574 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-token" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:18.234645 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.233968 4991 scope.go:122] "RemoveContainer" containerID="f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25" Oct 30 22:49:18.234740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.234697 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-kube-api-access-zdrtq" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "kube-api-access-zdrtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:18.235656 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.235615 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-envoy" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:18.236208 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.236187 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-podinfo" pod "3b242b11-a4fb-47d9-a92e-1233ae478508" (UID: "3b242b11-a4fb-47d9-a92e-1233ae478508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:49:18.272650 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.272633 4991 scope.go:122] "RemoveContainer" containerID="79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065" Oct 30 22:49:18.272928 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:18.272906 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065\": container with ID starting with 79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065 not found: ID does not exist" containerID="79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065" Oct 30 22:49:18.273013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.272940 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065"} err="failed to get container status \"79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065\": rpc error: code = NotFound desc = could not find container \"79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065\": container with ID starting with 79333a894ec8948e1015aa0cb33ead9373f0ae0ccd8caa8f4514103fb137f065 not found: ID does not exist" Oct 30 22:49:18.273013 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.272963 4991 scope.go:122] "RemoveContainer" containerID="f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25" Oct 30 22:49:18.273216 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:18.273198 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25\": container with ID starting with f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25 not found: ID does not exist" containerID="f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25" Oct 30 22:49:18.273260 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.273222 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25"} err="failed to get container status \"f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25\": rpc error: code = NotFound desc = could not find container \"f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25\": container with ID starting with f9ea1c0fa2faa0fb07fedd501290e3085417e20a274addd2404717b557431a25 not found: ID does not exist" Oct 30 22:49:18.333160 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333130 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.333160 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333158 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.333307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333174 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrtq\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-kube-api-access-zdrtq\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.333307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333187 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.333307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333200 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.333307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333213 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.333307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333226 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b242b11-a4fb-47d9-a92e-1233ae478508-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.333307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.333241 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b242b11-a4fb-47d9-a92e-1233ae478508-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:18.565928 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.565889 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt"] Oct 30 22:49:18.566148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.566123 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" containerID="cri-o://6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184" gracePeriod=30 Oct 30 22:49:18.640556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.640525 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg"] Oct 30 22:49:18.645354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:18.645296 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-031eee-openshift-default-55d57b55c7-8qllg"] Oct 30 22:49:19.876981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:19.876950 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" path="/var/lib/kubelet/pods/3b242b11-a4fb-47d9-a92e-1233ae478508/volumes" Oct 30 22:49:19.941081 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:19.941051 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9"] Oct 30 22:49:21.318252 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.318184 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27"] Oct 30 22:49:21.319087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319063 4991 cpu_manager.go:414] "containerMap: removing container" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" Oct 30 22:49:21.319175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319088 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" Oct 30 22:49:21.319175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319135 4991 cpu_manager.go:414] "containerMap: removing container" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" Oct 30 22:49:21.319175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319143 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" Oct 30 22:49:21.319175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319158 4991 cpu_manager.go:414] "containerMap: removing container" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" Oct 30 22:49:21.319175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319164 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" Oct 30 22:49:21.319469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319179 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2d51cf74-f581-4f09-9d39-c5d421be1776" containerName="maas-api" Oct 30 22:49:21.319469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319186 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d51cf74-f581-4f09-9d39-c5d421be1776" containerName="maas-api" Oct 30 22:49:21.319469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319363 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" Oct 30 22:49:21.319469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319386 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" Oct 30 22:49:21.319469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319404 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b242b11-a4fb-47d9-a92e-1233ae478508" containerName="istio-proxy" Oct 30 22:49:21.319469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.319417 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d51cf74-f581-4f09-9d39-c5d421be1776" containerName="maas-api" Oct 30 22:49:21.326167 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.326140 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.385416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.383686 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27"] Oct 30 22:49:21.400388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400361 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-token\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.400545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400403 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-envoy\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.400545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400424 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istiod-ca-cert\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.400545 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400506 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25nkg\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-kube-api-access-25nkg\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.400736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400563 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-credential-socket\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.400736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400598 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-data\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.400736 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400707 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-certs\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.400984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.400952 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-podinfo\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.401054 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.401008 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-socket\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.424246 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.424224 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9626a6-openshift-default-dockercfg-m277k\"" Oct 30 22:49:21.502618 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502583 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-podinfo\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.502818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502634 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-socket\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.502818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502670 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-token\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.502818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502711 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-envoy\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.502818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502733 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istiod-ca-cert\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502845 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25nkg\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-kube-api-access-25nkg\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502917 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-credential-socket\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.502949 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-data\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503060 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.503021 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-certs\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503278 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.503110 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-socket\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.503479 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-certs\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.503506 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-credential-socket\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.503575 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-data\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.503799 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.503777 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istiod-ca-cert\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.505587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.505563 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-envoy\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.505831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.505810 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-podinfo\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.560717 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.560693 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-token\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.560928 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.560907 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25nkg\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-kube-api-access-25nkg\") pod \"e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:21.663070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:21.663005 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:22.186046 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:22.186012 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27"] Oct 30 22:49:22.234392 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:49:22.234352 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a6c4db_a0b8_442c_90f4_52f3cfe31a29.slice/crio-b0e309079e76dd691a2b773b681de00fb0c5c3bf15468a48b0fe09324a98fc87 WatchSource:0}: Error finding container b0e309079e76dd691a2b773b681de00fb0c5c3bf15468a48b0fe09324a98fc87: Status 404 returned error can't find the container with id b0e309079e76dd691a2b773b681de00fb0c5c3bf15468a48b0fe09324a98fc87 Oct 30 22:49:22.237238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:22.237205 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:49:22.237374 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:22.237290 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:49:22.237450 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:22.237373 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:49:22.288354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:22.288291 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" event={"ID":"07a6c4db-a0b8-442c-90f4-52f3cfe31a29","Type":"ContainerStarted","Data":"b0e309079e76dd691a2b773b681de00fb0c5c3bf15468a48b0fe09324a98fc87"} Oct 30 22:49:22.487191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:22.487153 4991 patch_prober.go:29] interesting pod/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:49:22.487588 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:22.487216 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:49:23.297131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:23.297096 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" event={"ID":"07a6c4db-a0b8-442c-90f4-52f3cfe31a29","Type":"ContainerStarted","Data":"06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab"} Oct 30 22:49:23.462548 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:23.462474 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" podStartSLOduration=2.462455988 podStartE2EDuration="2.462455988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:23.458463218 +0000 UTC m=+3783.757873102" watchObservedRunningTime="2026-10-30 22:49:23.462455988 +0000 UTC m=+3783.761865868" Oct 30 22:49:23.663899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:23.663864 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:23.669961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:23.669936 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:23.863574 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:23.863515 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:49:24.036866 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.036834 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-envoy\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.036887 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-credential-socket\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.036908 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-data\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.036938 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-certs\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.036961 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-token\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-token\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.036988 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istiod-ca-cert\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037042 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-kube-api-access-kq7rh\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-kube-api-access-kq7rh\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037209 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-podinfo\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037216 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-credential-socket" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:24.037429 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037262 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-certs" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:24.037733 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037416 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-socket\") pod \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\" (UID: \"df1440bd-b906-43d9-9a0b-9a2ed036abe7\") " Oct 30 22:49:24.037832 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037671 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-socket" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:24.037832 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037688 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istiod-ca-cert" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:24.038033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.037999 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-data" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:24.039743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.039711 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.039743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.039740 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.039920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.039754 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.039920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.039769 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.039920 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.039788 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.040070 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.040012 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-podinfo" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:49:24.040194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.040173 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-envoy" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:24.040302 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.040284 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-kube-api-access-kq7rh" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "kube-api-access-kq7rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:24.041283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.041259 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-token" pod "df1440bd-b906-43d9-9a0b-9a2ed036abe7" (UID: "df1440bd-b906-43d9-9a0b-9a2ed036abe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:24.141408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.141331 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.141408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.141362 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.141408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.141373 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.141408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.141383 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7rh\" (UniqueName: \"kubernetes.io/projected/df1440bd-b906-43d9-9a0b-9a2ed036abe7-kube-api-access-kq7rh\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:24.305380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.305345 4991 generic.go:356] "Generic (PLEG): container finished" podID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerID="6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184" exitCode=0 Oct 30 22:49:24.305567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.305385 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" Oct 30 22:49:24.305567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.305425 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" event={"ID":"df1440bd-b906-43d9-9a0b-9a2ed036abe7","Type":"ContainerDied","Data":"6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184"} Oct 30 22:49:24.305567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.305497 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt" event={"ID":"df1440bd-b906-43d9-9a0b-9a2ed036abe7","Type":"ContainerDied","Data":"8b03d94c70669614d1f53c42a809437b3fea49bccfa0937e5741573d08344b30"} Oct 30 22:49:24.305567 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.305528 4991 scope.go:122] "RemoveContainer" containerID="6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184" Oct 30 22:49:24.306043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.305960 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:24.306906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.306881 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:49:24.316976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.316959 4991 scope.go:122] "RemoveContainer" containerID="40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677" Oct 30 22:49:24.328090 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.327939 4991 scope.go:122] "RemoveContainer" containerID="6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184" Oct 30 22:49:24.328266 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:24.328247 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184\": container with ID starting with 6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184 not found: ID does not exist" containerID="6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184" Oct 30 22:49:24.328386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.328274 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184"} err="failed to get container status \"6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184\": rpc error: code = NotFound desc = could not find container \"6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184\": container with ID starting with 6ff8779e6b073bec73ee4242b4aed087ee9c8fbce07406927498b50b5e4b8184 not found: ID does not exist" Oct 30 22:49:24.328386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.328291 4991 scope.go:122] "RemoveContainer" containerID="40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677" Oct 30 22:49:24.329043 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:24.329025 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677\": container with ID starting with 40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677 not found: ID does not exist" containerID="40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677" Oct 30 22:49:24.329125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.329049 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677"} err="failed to get container status \"40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677\": rpc error: code = NotFound desc = could not find container \"40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677\": container with ID starting with 40e059af3c060c0b635d9ebcdae4885ba98d1026a41a6e0b9737195f852c2677 not found: ID does not exist" Oct 30 22:49:24.463141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.463054 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt"] Oct 30 22:49:24.509604 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:24.509575 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-dcde03-openshift-default-bd5cc4777-v8tjt"] Oct 30 22:49:25.877618 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:25.877587 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" path="/var/lib/kubelet/pods/df1440bd-b906-43d9-9a0b-9a2ed036abe7/volumes" Oct 30 22:49:34.962532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:34.962466 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" containerID="cri-o://46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0" gracePeriod=45 Oct 30 22:49:35.232647 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.232623 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:49:35.353769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.353742 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-tmp-dir\") pod \"e45e44a5-3029-43ff-a870-4190e9a041ec\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " Oct 30 22:49:35.353956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.353812 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45e44a5-3029-43ff-a870-4190e9a041ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/e45e44a5-3029-43ff-a870-4190e9a041ec-tls-certs\") pod \"e45e44a5-3029-43ff-a870-4190e9a041ec\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " Oct 30 22:49:35.353956 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.353879 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-model-cache\") pod \"e45e44a5-3029-43ff-a870-4190e9a041ec\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " Oct 30 22:49:35.354079 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.353973 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-dshm\") pod \"e45e44a5-3029-43ff-a870-4190e9a041ec\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " Oct 30 22:49:35.354079 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.354043 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-home\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-home\") pod \"e45e44a5-3029-43ff-a870-4190e9a041ec\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " Oct 30 22:49:35.354182 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.354110 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45e44a5-3029-43ff-a870-4190e9a041ec-kube-api-access-fmn2v\" (UniqueName: \"kubernetes.io/projected/e45e44a5-3029-43ff-a870-4190e9a041ec-kube-api-access-fmn2v\") pod \"e45e44a5-3029-43ff-a870-4190e9a041ec\" (UID: \"e45e44a5-3029-43ff-a870-4190e9a041ec\") " Oct 30 22:49:35.358482 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.357534 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45e44a5-3029-43ff-a870-4190e9a041ec-kube-api-access-fmn2v" pod "e45e44a5-3029-43ff-a870-4190e9a041ec" (UID: "e45e44a5-3029-43ff-a870-4190e9a041ec"). InnerVolumeSpecName "kube-api-access-fmn2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:35.365248 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.365223 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45e44a5-3029-43ff-a870-4190e9a041ec-tls-certs" pod "e45e44a5-3029-43ff-a870-4190e9a041ec" (UID: "e45e44a5-3029-43ff-a870-4190e9a041ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:35.365480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.365461 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-model-cache" pod "e45e44a5-3029-43ff-a870-4190e9a041ec" (UID: "e45e44a5-3029-43ff-a870-4190e9a041ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.365969 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.365947 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-home" pod "e45e44a5-3029-43ff-a870-4190e9a041ec" (UID: "e45e44a5-3029-43ff-a870-4190e9a041ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.367363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.367312 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-dshm" pod "e45e44a5-3029-43ff-a870-4190e9a041ec" (UID: "e45e44a5-3029-43ff-a870-4190e9a041ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.368559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.368529 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-tmp-dir" pod "e45e44a5-3029-43ff-a870-4190e9a041ec" (UID: "e45e44a5-3029-43ff-a870-4190e9a041ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.396125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.396096 4991 generic.go:356] "Generic (PLEG): container finished" podID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerID="46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0" exitCode=0 Oct 30 22:49:35.396243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.396142 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" event={"ID":"e45e44a5-3029-43ff-a870-4190e9a041ec","Type":"ContainerDied","Data":"46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0"} Oct 30 22:49:35.396243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.396170 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" event={"ID":"e45e44a5-3029-43ff-a870-4190e9a041ec","Type":"ContainerDied","Data":"6ff336ffcbd9213b75261b639eec5f30aa46631fd52afc72e5790e60a83a11f3"} Oct 30 22:49:35.396243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.396186 4991 scope.go:122] "RemoveContainer" containerID="46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0" Oct 30 22:49:35.396456 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.396306 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" Oct 30 22:49:35.408858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.408841 4991 scope.go:122] "RemoveContainer" containerID="46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0" Oct 30 22:49:35.409178 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:35.409158 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0\": container with ID starting with 46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0 not found: ID does not exist" containerID="46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0" Oct 30 22:49:35.409238 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.409187 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0"} err="failed to get container status \"46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0\": rpc error: code = NotFound desc = could not find container \"46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0\": container with ID starting with 46a923e2a94f0a0083f3343d75867b24b626fd86af454a35236347afd52405a0 not found: ID does not exist" Oct 30 22:49:35.456394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.456230 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmn2v\" (UniqueName: \"kubernetes.io/projected/e45e44a5-3029-43ff-a870-4190e9a041ec-kube-api-access-fmn2v\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.456394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.456268 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.456394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.456282 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45e44a5-3029-43ff-a870-4190e9a041ec-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.456394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.456296 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.456394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.456310 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.456394 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.456345 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45e44a5-3029-43ff-a870-4190e9a041ec-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.564418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.564339 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9"] Oct 30 22:49:35.571182 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.571156 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9"] Oct 30 22:49:35.876833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:35.876753 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" path="/var/lib/kubelet/pods/e45e44a5-3029-43ff-a870-4190e9a041ec/volumes" Oct 30 22:49:36.219993 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:36.219912 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-2-w1-e65019-kserve-76f9765b95-crck9" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:49:47.912829 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:49:47.912787 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:49:53.247859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.247819 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk"] Oct 30 22:49:53.248388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.248036 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" podUID="f946456d-2550-4e3f-adce-ff9fe63789cb" containerName="maas-api" containerID="cri-o://75851a0b1734799aa447db3bae09e3c76a2b54c0389194c2880b4963eacf1b4d" gracePeriod=30 Oct 30 22:49:53.672640 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.672444 4991 generic.go:356] "Generic (PLEG): container finished" podID="f946456d-2550-4e3f-adce-ff9fe63789cb" containerID="75851a0b1734799aa447db3bae09e3c76a2b54c0389194c2880b4963eacf1b4d" exitCode=0 Oct 30 22:49:53.673150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.673006 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" event={"ID":"f946456d-2550-4e3f-adce-ff9fe63789cb","Type":"ContainerDied","Data":"75851a0b1734799aa447db3bae09e3c76a2b54c0389194c2880b4963eacf1b4d"} Oct 30 22:49:53.683915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.683512 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:49:53.842904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.842815 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f946456d-2550-4e3f-adce-ff9fe63789cb-kube-api-access-jvk66\" (UniqueName: \"kubernetes.io/projected/f946456d-2550-4e3f-adce-ff9fe63789cb-kube-api-access-jvk66\") pod \"f946456d-2550-4e3f-adce-ff9fe63789cb\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " Oct 30 22:49:53.843119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.842916 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f946456d-2550-4e3f-adce-ff9fe63789cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f946456d-2550-4e3f-adce-ff9fe63789cb-openshift-service-ca\") pod \"f946456d-2550-4e3f-adce-ff9fe63789cb\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " Oct 30 22:49:53.843119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.843094 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-metrics-certs\") pod \"f946456d-2550-4e3f-adce-ff9fe63789cb\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " Oct 30 22:49:53.843284 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.843267 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls\") pod \"f946456d-2550-4e3f-adce-ff9fe63789cb\" (UID: \"f946456d-2550-4e3f-adce-ff9fe63789cb\") " Oct 30 22:49:53.843362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.843278 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f946456d-2550-4e3f-adce-ff9fe63789cb-openshift-service-ca" pod "f946456d-2550-4e3f-adce-ff9fe63789cb" (UID: "f946456d-2550-4e3f-adce-ff9fe63789cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:53.843875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.843847 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f946456d-2550-4e3f-adce-ff9fe63789cb-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:53.846415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.846389 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-metrics-certs" pod "f946456d-2550-4e3f-adce-ff9fe63789cb" (UID: "f946456d-2550-4e3f-adce-ff9fe63789cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:53.847485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.847459 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f946456d-2550-4e3f-adce-ff9fe63789cb-kube-api-access-jvk66" pod "f946456d-2550-4e3f-adce-ff9fe63789cb" (UID: "f946456d-2550-4e3f-adce-ff9fe63789cb"). InnerVolumeSpecName "kube-api-access-jvk66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:53.850106 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.848428 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls" pod "f946456d-2550-4e3f-adce-ff9fe63789cb" (UID: "f946456d-2550-4e3f-adce-ff9fe63789cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:53.945764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.945727 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:53.945764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.945765 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvk66\" (UniqueName: \"kubernetes.io/projected/f946456d-2550-4e3f-adce-ff9fe63789cb-kube-api-access-jvk66\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:53.945976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:53.945781 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f946456d-2550-4e3f-adce-ff9fe63789cb-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:54.681498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:54.681464 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" Oct 30 22:49:54.681976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:54.681502 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk" event={"ID":"f946456d-2550-4e3f-adce-ff9fe63789cb","Type":"ContainerDied","Data":"872b54f4dc7d7cb983e593fe64b36aabfc0a4fba18f9072e5f986324524a1597"} Oct 30 22:49:54.681976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:54.681573 4991 scope.go:122] "RemoveContainer" containerID="75851a0b1734799aa447db3bae09e3c76a2b54c0389194c2880b4963eacf1b4d" Oct 30 22:49:54.840455 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:54.840422 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk"] Oct 30 22:49:54.885109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:54.885079 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b829bc-6d54bbf85d-db4fk"] Oct 30 22:49:55.875872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:49:55.875839 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f946456d-2550-4e3f-adce-ff9fe63789cb" path="/var/lib/kubelet/pods/f946456d-2550-4e3f-adce-ff9fe63789cb/volumes" Oct 30 22:50:02.825400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:02.824425 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms"] Oct 30 22:50:02.825400 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:02.824741 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" containerID="cri-o://dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3" gracePeriod=30 Oct 30 22:50:03.109442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:03.109349 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:50:03.109442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:03.109431 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:50:08.285867 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.285840 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:50:08.381022 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.380956 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-credential-socket\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381022 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381000 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-data\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381031 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-kube-api-access-65gwn\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-kube-api-access-65gwn\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381063 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-envoy\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381141 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-token\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381223 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381211 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-socket\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381269 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-data" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:08.381472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381274 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-certs\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381288 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-credential-socket" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:08.381472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381351 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-podinfo\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381422 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istiod-ca-cert\") pod \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\" (UID: \"2ad2290c-3dae-4d9a-b985-8820a0611b5c\") " Oct 30 22:50:08.381743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381499 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-certs" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:08.381743 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381675 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-socket" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:08.381868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.381788 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istiod-ca-cert" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:50:08.382269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.382058 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.382269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.382075 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.382269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.382084 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.382269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.382095 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.382269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.382109 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.383581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.383546 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-kube-api-access-65gwn" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "kube-api-access-65gwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:08.383761 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.383741 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-envoy" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:08.383897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.383882 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-token" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:08.383945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.383897 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-podinfo" pod "2ad2290c-3dae-4d9a-b985-8820a0611b5c" (UID: "2ad2290c-3dae-4d9a-b985-8820a0611b5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:50:08.482895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.482859 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.482895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.482890 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65gwn\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-kube-api-access-65gwn\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.483082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.482905 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.483082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.482918 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ad2290c-3dae-4d9a-b985-8820a0611b5c-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:08.821778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.821742 4991 generic.go:356] "Generic (PLEG): container finished" podID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerID="dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3" exitCode=0 Oct 30 22:50:08.821977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.821805 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" Oct 30 22:50:08.821977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.821828 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" event={"ID":"2ad2290c-3dae-4d9a-b985-8820a0611b5c","Type":"ContainerDied","Data":"dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3"} Oct 30 22:50:08.821977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.821899 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms" event={"ID":"2ad2290c-3dae-4d9a-b985-8820a0611b5c","Type":"ContainerDied","Data":"dc5572ca6be86890153cbd89f38ee170ddee8713c165234dcb4ce8cca9d63cd7"} Oct 30 22:50:08.821977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.821931 4991 scope.go:122] "RemoveContainer" containerID="dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3" Oct 30 22:50:08.833651 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.833633 4991 scope.go:122] "RemoveContainer" containerID="15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8" Oct 30 22:50:08.842918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.842898 4991 scope.go:122] "RemoveContainer" containerID="dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3" Oct 30 22:50:08.843227 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:50:08.843205 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3\": container with ID starting with dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3 not found: ID does not exist" containerID="dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3" Oct 30 22:50:08.843296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.843235 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3"} err="failed to get container status \"dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3\": rpc error: code = NotFound desc = could not find container \"dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3\": container with ID starting with dd8211f1ef3f6c138b14e41db37dd72eb2fe95fe5afa6ec55aeb6cc5fb8358c3 not found: ID does not exist" Oct 30 22:50:08.843296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.843253 4991 scope.go:122] "RemoveContainer" containerID="15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8" Oct 30 22:50:08.843511 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:50:08.843490 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8\": container with ID starting with 15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8 not found: ID does not exist" containerID="15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8" Oct 30 22:50:08.843553 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.843516 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8"} err="failed to get container status \"15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8\": rpc error: code = NotFound desc = could not find container \"15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8\": container with ID starting with 15a4a877bdc7bc89033f15d9bbfbcce98f4c11f58b03c09cceafc0a7a497fff8 not found: ID does not exist" Oct 30 22:50:08.980665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:08.980632 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms"] Oct 30 22:50:09.029071 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:09.029032 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b829bc-openshift-default-586c94979c-x52ms"] Oct 30 22:50:09.878107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:09.878074 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" path="/var/lib/kubelet/pods/2ad2290c-3dae-4d9a-b985-8820a0611b5c/volumes" Oct 30 22:50:17.913175 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:50:17.913138 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:50:47.913687 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:50:47.913624 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:50:59.696406 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:59.696367 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2"] Oct 30 22:50:59.697376 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:59.697334 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" podUID="69284587-4eeb-462e-938c-e3b7f3bb85fb" containerName="maas-api" containerID="cri-o://2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b" gracePeriod=30 Oct 30 22:50:59.996851 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:50:59.996823 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:51:00.034130 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.034104 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls\") pod \"69284587-4eeb-462e-938c-e3b7f3bb85fb\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " Oct 30 22:51:00.034242 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.034203 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-metrics-certs\") pod \"69284587-4eeb-462e-938c-e3b7f3bb85fb\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " Oct 30 22:51:00.034437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.034412 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69284587-4eeb-462e-938c-e3b7f3bb85fb-kube-api-access-mm75v\" (UniqueName: \"kubernetes.io/projected/69284587-4eeb-462e-938c-e3b7f3bb85fb-kube-api-access-mm75v\") pod \"69284587-4eeb-462e-938c-e3b7f3bb85fb\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " Oct 30 22:51:00.034939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.034468 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69284587-4eeb-462e-938c-e3b7f3bb85fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69284587-4eeb-462e-938c-e3b7f3bb85fb-openshift-service-ca\") pod \"69284587-4eeb-462e-938c-e3b7f3bb85fb\" (UID: \"69284587-4eeb-462e-938c-e3b7f3bb85fb\") " Oct 30 22:51:00.034939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.034858 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69284587-4eeb-462e-938c-e3b7f3bb85fb-openshift-service-ca" pod "69284587-4eeb-462e-938c-e3b7f3bb85fb" (UID: "69284587-4eeb-462e-938c-e3b7f3bb85fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:00.035080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.035038 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69284587-4eeb-462e-938c-e3b7f3bb85fb-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:00.036804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.036772 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-metrics-certs" pod "69284587-4eeb-462e-938c-e3b7f3bb85fb" (UID: "69284587-4eeb-462e-938c-e3b7f3bb85fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:00.036904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.036806 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls" pod "69284587-4eeb-462e-938c-e3b7f3bb85fb" (UID: "69284587-4eeb-462e-938c-e3b7f3bb85fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:00.036904 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.036849 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69284587-4eeb-462e-938c-e3b7f3bb85fb-kube-api-access-mm75v" pod "69284587-4eeb-462e-938c-e3b7f3bb85fb" (UID: "69284587-4eeb-462e-938c-e3b7f3bb85fb"). InnerVolumeSpecName "kube-api-access-mm75v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:00.135985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.135946 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm75v\" (UniqueName: \"kubernetes.io/projected/69284587-4eeb-462e-938c-e3b7f3bb85fb-kube-api-access-mm75v\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:00.135985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.135980 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:00.135985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.135992 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69284587-4eeb-462e-938c-e3b7f3bb85fb-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:00.201193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.201158 4991 generic.go:356] "Generic (PLEG): container finished" podID="69284587-4eeb-462e-938c-e3b7f3bb85fb" containerID="2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b" exitCode=0 Oct 30 22:51:00.201378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.201202 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" Oct 30 22:51:00.201378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.201251 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" event={"ID":"69284587-4eeb-462e-938c-e3b7f3bb85fb","Type":"ContainerDied","Data":"2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b"} Oct 30 22:51:00.201378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.201303 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2" event={"ID":"69284587-4eeb-462e-938c-e3b7f3bb85fb","Type":"ContainerDied","Data":"7911ae58a665a4a0edc1fd8c201c0df6044534913c7b4a47980f434618a5b548"} Oct 30 22:51:00.201378 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.201338 4991 scope.go:122] "RemoveContainer" containerID="2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b" Oct 30 22:51:00.212189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.212174 4991 scope.go:122] "RemoveContainer" containerID="2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b" Oct 30 22:51:00.212464 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:51:00.212447 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b\": container with ID starting with 2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b not found: ID does not exist" containerID="2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b" Oct 30 22:51:00.212517 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.212474 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b"} err="failed to get container status \"2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b\": rpc error: code = NotFound desc = could not find container \"2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b\": container with ID starting with 2d216737ff68f35f404c457691c7141e9daa86b635a59c73e7c6856de467784b not found: ID does not exist" Oct 30 22:51:00.359082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.359041 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2"] Oct 30 22:51:00.365294 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:00.365263 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9a60dd-77b6df454-pzxr2"] Oct 30 22:51:01.876293 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:01.876263 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69284587-4eeb-462e-938c-e3b7f3bb85fb" path="/var/lib/kubelet/pods/69284587-4eeb-462e-938c-e3b7f3bb85fb/volumes" Oct 30 22:51:17.911257 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:51:17.911215 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:51:20.577721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:20.577693 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:51:20.589126 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:20.589104 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:51:33.450216 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:33.450176 4991 generic.go:356] "Generic (PLEG): container finished" podID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerID="06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab" exitCode=0 Oct 30 22:51:33.450706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:33.450249 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" event={"ID":"07a6c4db-a0b8-442c-90f4-52f3cfe31a29","Type":"ContainerDied","Data":"06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab"} Oct 30 22:51:33.450706 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:33.450598 4991 scope.go:122] "RemoveContainer" containerID="06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab" Oct 30 22:51:33.452065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:33.452039 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:51:33.452147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:33.452111 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:51:33.452185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:33.452146 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:51:34.100782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.100749 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6"] Oct 30 22:51:34.100958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.100940 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" podUID="dbfd066a-4b0b-4929-904f-aaf4d73d9dab" containerName="maas-api" containerID="cri-o://03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19" gracePeriod=30 Oct 30 22:51:34.500669 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.498868 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" event={"ID":"07a6c4db-a0b8-442c-90f4-52f3cfe31a29","Type":"ContainerStarted","Data":"03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376"} Oct 30 22:51:34.505760 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.505733 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:51:34.506353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.506329 4991 generic.go:356] "Generic (PLEG): container finished" podID="dbfd066a-4b0b-4929-904f-aaf4d73d9dab" containerID="03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19" exitCode=0 Oct 30 22:51:34.506581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.506505 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" event={"ID":"dbfd066a-4b0b-4929-904f-aaf4d73d9dab","Type":"ContainerDied","Data":"03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19"} Oct 30 22:51:34.506731 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.506718 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" event={"ID":"dbfd066a-4b0b-4929-904f-aaf4d73d9dab","Type":"ContainerDied","Data":"90583c416ed26a1ebede587fa522431d54f9ed11d3f9eba6a97e922d178abb8d"} Oct 30 22:51:34.506843 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.506832 4991 scope.go:122] "RemoveContainer" containerID="03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19" Oct 30 22:51:34.556532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.556498 4991 scope.go:122] "RemoveContainer" containerID="03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19" Oct 30 22:51:34.556941 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:51:34.556916 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19\": container with ID starting with 03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19 not found: ID does not exist" containerID="03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19" Oct 30 22:51:34.557051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.556948 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19"} err="failed to get container status \"03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19\": rpc error: code = NotFound desc = could not find container \"03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19\": container with ID starting with 03312408f922e72d9d09815fb6622248565ff6cec037784bce7783662cb55c19 not found: ID does not exist" Oct 30 22:51:34.656881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.656780 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-maas-api-tls\") pod \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " Oct 30 22:51:34.656881 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.656859 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-openshift-service-ca\") pod \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " Oct 30 22:51:34.657140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.656986 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-metrics-certs\") pod \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " Oct 30 22:51:34.657140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.657050 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-kube-api-access-vl7t7\" (UniqueName: \"kubernetes.io/projected/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-kube-api-access-vl7t7\") pod \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\" (UID: \"dbfd066a-4b0b-4929-904f-aaf4d73d9dab\") " Oct 30 22:51:34.657986 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.657950 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-openshift-service-ca" pod "dbfd066a-4b0b-4929-904f-aaf4d73d9dab" (UID: "dbfd066a-4b0b-4929-904f-aaf4d73d9dab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:34.659817 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.659783 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-metrics-certs" pod "dbfd066a-4b0b-4929-904f-aaf4d73d9dab" (UID: "dbfd066a-4b0b-4929-904f-aaf4d73d9dab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:34.659951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.659823 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-kube-api-access-vl7t7" pod "dbfd066a-4b0b-4929-904f-aaf4d73d9dab" (UID: "dbfd066a-4b0b-4929-904f-aaf4d73d9dab"). InnerVolumeSpecName "kube-api-access-vl7t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:34.660019 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.660000 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-maas-api-tls" pod "dbfd066a-4b0b-4929-904f-aaf4d73d9dab" (UID: "dbfd066a-4b0b-4929-904f-aaf4d73d9dab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:34.663180 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.663154 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:51:34.664072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.664047 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 22:51:34.664173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.664104 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 22:51:34.759191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.759159 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:34.759191 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.759189 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:34.759434 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.759201 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:34.759434 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:34.759210 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl7t7\" (UniqueName: \"kubernetes.io/projected/dbfd066a-4b0b-4929-904f-aaf4d73d9dab-kube-api-access-vl7t7\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:35.514148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:35.514118 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6" Oct 30 22:51:35.663965 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:35.663915 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 22:51:35.664161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:35.663979 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 22:51:35.687235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:35.687205 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6"] Oct 30 22:51:35.718153 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:35.718125 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f4e241-6987968c95-n8kw6"] Oct 30 22:51:35.877199 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:35.877124 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfd066a-4b0b-4929-904f-aaf4d73d9dab" path="/var/lib/kubelet/pods/dbfd066a-4b0b-4929-904f-aaf4d73d9dab/volumes" Oct 30 22:51:36.663646 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:36.663606 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 22:51:36.664093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:36.663672 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 22:51:37.664010 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:37.663966 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 22:51:37.664547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:37.664031 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 22:51:38.663894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:38.663854 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 22:51:38.664106 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:38.663926 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 22:51:39.306151 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:39.306116 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:51:39.737446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:39.737417 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:51:39.739150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:39.739120 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:51:43.547638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:43.547603 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x"] Oct 30 22:51:43.548024 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:43.547824 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" containerID="cri-o://ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af" gracePeriod=30 Oct 30 22:51:47.655516 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:47.655475 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:51:47.655981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:47.655531 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:51:47.913860 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:51:47.913778 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:51:48.819031 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.818976 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:51:48.933511 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933476 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-credential-socket\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.933673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933530 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-envoy\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.933673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933560 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-certs\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.933673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933597 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7669c06-6d57-4547-b94b-9e5f07a55a36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7669c06-6d57-4547-b94b-9e5f07a55a36-istiod-ca-cert\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.933673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933625 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-podinfo\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.933673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933658 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-data\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.933976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933702 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-credential-socket" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:48.933976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933868 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-socket\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.933976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933970 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-data" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:48.934147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.933993 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-kube-api-access-fvb2l\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-kube-api-access-fvb2l\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.934147 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934048 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-token\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-token\") pod \"c7669c06-6d57-4547-b94b-9e5f07a55a36\" (UID: \"c7669c06-6d57-4547-b94b-9e5f07a55a36\") " Oct 30 22:51:48.934256 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934182 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-socket" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:48.934256 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934089 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7669c06-6d57-4547-b94b-9e5f07a55a36-istiod-ca-cert" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:48.934386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934274 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-certs" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:48.935203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934683 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:48.935203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934711 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:48.935203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934728 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7669c06-6d57-4547-b94b-9e5f07a55a36-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:48.935203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934743 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:48.935203 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.934760 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:48.937544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.937517 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-envoy" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:48.937644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.937521 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-podinfo" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:51:48.937644 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.937565 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-token" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:48.938637 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:48.938614 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-kube-api-access-fvb2l" pod "c7669c06-6d57-4547-b94b-9e5f07a55a36" (UID: "c7669c06-6d57-4547-b94b-9e5f07a55a36"). InnerVolumeSpecName "kube-api-access-fvb2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:49.035475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.035437 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvb2l\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-kube-api-access-fvb2l\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.035475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.035463 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.035475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.035474 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.035475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.035484 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7669c06-6d57-4547-b94b-9e5f07a55a36-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.648498 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.648467 4991 generic.go:356] "Generic (PLEG): container finished" podID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerID="ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af" exitCode=0 Oct 30 22:51:49.648672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.648516 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" Oct 30 22:51:49.648672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.648553 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" event={"ID":"c7669c06-6d57-4547-b94b-9e5f07a55a36","Type":"ContainerDied","Data":"ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af"} Oct 30 22:51:49.648672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.648609 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x" event={"ID":"c7669c06-6d57-4547-b94b-9e5f07a55a36","Type":"ContainerDied","Data":"f010dbbea166493042a602614379c10fbf9004fcd06c22f68759963ced4fc623"} Oct 30 22:51:49.648672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.648639 4991 scope.go:122] "RemoveContainer" containerID="ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af" Oct 30 22:51:49.660041 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.660022 4991 scope.go:122] "RemoveContainer" containerID="e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302" Oct 30 22:51:49.669683 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.669666 4991 scope.go:122] "RemoveContainer" containerID="ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af" Oct 30 22:51:49.669942 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:51:49.669925 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af\": container with ID starting with ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af not found: ID does not exist" containerID="ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af" Oct 30 22:51:49.669997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.669948 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af"} err="failed to get container status \"ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af\": rpc error: code = NotFound desc = could not find container \"ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af\": container with ID starting with ac7ab4f5d55c2d70b00765abe5a5a4edd669d33862f7b9ed7a95704fd97a82af not found: ID does not exist" Oct 30 22:51:49.669997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.669967 4991 scope.go:122] "RemoveContainer" containerID="e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302" Oct 30 22:51:49.670195 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:51:49.670177 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302\": container with ID starting with e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302 not found: ID does not exist" containerID="e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302" Oct 30 22:51:49.670239 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.670202 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302"} err="failed to get container status \"e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302\": rpc error: code = NotFound desc = could not find container \"e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302\": container with ID starting with e3fcfcff0eb9258c4502d2d77c9d73766688a11b732bee90552433e52bd5c302 not found: ID does not exist" Oct 30 22:51:49.876580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.876554 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x"] Oct 30 22:51:49.883618 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:49.883595 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f4e241-openshift-default-66676d5865-nz94x"] Oct 30 22:51:51.876781 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:51:51.876737 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" path="/var/lib/kubelet/pods/c7669c06-6d57-4547-b94b-9e5f07a55a36/volumes" Oct 30 22:52:09.170597 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.170561 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5"] Oct 30 22:52:09.171134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.170795 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" podUID="895dfcd2-d9dd-4120-98e9-12f581eee595" containerName="maas-api" containerID="cri-o://d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524" gracePeriod=30 Oct 30 22:52:09.461177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.461155 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:52:09.598556 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.598522 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895dfcd2-d9dd-4120-98e9-12f581eee595-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/895dfcd2-d9dd-4120-98e9-12f581eee595-openshift-service-ca\") pod \"895dfcd2-d9dd-4120-98e9-12f581eee595\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " Oct 30 22:52:09.598729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.598675 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-metrics-certs\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-metrics-certs\") pod \"895dfcd2-d9dd-4120-98e9-12f581eee595\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " Oct 30 22:52:09.598729 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.598719 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895dfcd2-d9dd-4120-98e9-12f581eee595-kube-api-access-vfmch\" (UniqueName: \"kubernetes.io/projected/895dfcd2-d9dd-4120-98e9-12f581eee595-kube-api-access-vfmch\") pod \"895dfcd2-d9dd-4120-98e9-12f581eee595\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " Oct 30 22:52:09.598917 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.598895 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls\") pod \"895dfcd2-d9dd-4120-98e9-12f581eee595\" (UID: \"895dfcd2-d9dd-4120-98e9-12f581eee595\") " Oct 30 22:52:09.599035 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.598929 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895dfcd2-d9dd-4120-98e9-12f581eee595-openshift-service-ca" pod "895dfcd2-d9dd-4120-98e9-12f581eee595" (UID: "895dfcd2-d9dd-4120-98e9-12f581eee595"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:09.599410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.599384 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/895dfcd2-d9dd-4120-98e9-12f581eee595-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:09.601539 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.601513 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895dfcd2-d9dd-4120-98e9-12f581eee595-kube-api-access-vfmch" pod "895dfcd2-d9dd-4120-98e9-12f581eee595" (UID: "895dfcd2-d9dd-4120-98e9-12f581eee595"). InnerVolumeSpecName "kube-api-access-vfmch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:09.601649 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.601589 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls" pod "895dfcd2-d9dd-4120-98e9-12f581eee595" (UID: "895dfcd2-d9dd-4120-98e9-12f581eee595"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:09.601753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.601729 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-metrics-certs" pod "895dfcd2-d9dd-4120-98e9-12f581eee595" (UID: "895dfcd2-d9dd-4120-98e9-12f581eee595"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:09.712255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.705779 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:09.712798 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.712773 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895dfcd2-d9dd-4120-98e9-12f581eee595-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:09.713016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.712981 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmch\" (UniqueName: \"kubernetes.io/projected/895dfcd2-d9dd-4120-98e9-12f581eee595-kube-api-access-vfmch\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:09.791263 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.791226 4991 generic.go:356] "Generic (PLEG): container finished" podID="895dfcd2-d9dd-4120-98e9-12f581eee595" containerID="d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524" exitCode=0 Oct 30 22:52:09.791471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.791272 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" event={"ID":"895dfcd2-d9dd-4120-98e9-12f581eee595","Type":"ContainerDied","Data":"d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524"} Oct 30 22:52:09.791471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.791276 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" Oct 30 22:52:09.791471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.791372 4991 scope.go:122] "RemoveContainer" containerID="d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524" Oct 30 22:52:09.791471 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.791359 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5" event={"ID":"895dfcd2-d9dd-4120-98e9-12f581eee595","Type":"ContainerDied","Data":"9c0bc4b50ba55319426f342b32b7497f81e0327eb3d3b59215e92a478e487207"} Oct 30 22:52:09.802971 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.802955 4991 scope.go:122] "RemoveContainer" containerID="d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524" Oct 30 22:52:09.803254 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:09.803229 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524\": container with ID starting with d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524 not found: ID does not exist" containerID="d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524" Oct 30 22:52:09.803381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.803259 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524"} err="failed to get container status \"d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524\": rpc error: code = NotFound desc = could not find container \"d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524\": container with ID starting with d0deaf939ad7b0682633df8743ca17eae966af36ec61aa2d6a501ff80c190524 not found: ID does not exist" Oct 30 22:52:09.962082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.962046 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5"] Oct 30 22:52:09.968575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:09.968498 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e65019-5c8cfd6698-wkxj5"] Oct 30 22:52:11.874622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:11.874591 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895dfcd2-d9dd-4120-98e9-12f581eee595" path="/var/lib/kubelet/pods/895dfcd2-d9dd-4120-98e9-12f581eee595/volumes" Oct 30 22:52:17.548885 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:17.548855 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c"] Oct 30 22:52:17.549354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:17.549064 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" containerID="cri-o://cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505" gracePeriod=30 Oct 30 22:52:17.911847 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:17.911764 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:52:19.392595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:19.392561 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp"] Oct 30 22:52:19.393112 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:19.392822 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" containerID="cri-o://7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8" gracePeriod=30 Oct 30 22:52:22.854355 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.854304 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:52:22.884422 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.884394 4991 generic.go:356] "Generic (PLEG): container finished" podID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerID="cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505" exitCode=0 Oct 30 22:52:22.884558 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.884439 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" Oct 30 22:52:22.884558 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.884477 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" event={"ID":"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7","Type":"ContainerDied","Data":"cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505"} Oct 30 22:52:22.884558 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.884537 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c" event={"ID":"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7","Type":"ContainerDied","Data":"dd374013d2235ce2db4b4ac966e5061743fd9a89bdbdc6accaf67378f2a4290f"} Oct 30 22:52:22.884740 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.884568 4991 scope.go:122] "RemoveContainer" containerID="cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505" Oct 30 22:52:22.894654 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.894636 4991 scope.go:122] "RemoveContainer" containerID="88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc" Oct 30 22:52:22.903704 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.903688 4991 scope.go:122] "RemoveContainer" containerID="cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505" Oct 30 22:52:22.903998 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:22.903958 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505\": container with ID starting with cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505 not found: ID does not exist" containerID="cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505" Oct 30 22:52:22.904051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.904007 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505"} err="failed to get container status \"cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505\": rpc error: code = NotFound desc = could not find container \"cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505\": container with ID starting with cfd912609296925b2e13c1e88ab47a44aa2c53133bb3dba5d0afb06f72316505 not found: ID does not exist" Oct 30 22:52:22.904051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.904025 4991 scope.go:122] "RemoveContainer" containerID="88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc" Oct 30 22:52:22.904232 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:22.904213 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc\": container with ID starting with 88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc not found: ID does not exist" containerID="88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc" Oct 30 22:52:22.904276 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.904238 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc"} err="failed to get container status \"88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc\": rpc error: code = NotFound desc = could not find container \"88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc\": container with ID starting with 88a6883ee8ceabddc36287c341abb78676987be26ea0e5d4e68048cfebd51fdc not found: ID does not exist" Oct 30 22:52:22.948684 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948661 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-token\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-token\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.948773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948721 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-podinfo\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.948818 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948800 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-envoy\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.948869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948821 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-credential-socket\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.948869 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948849 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-socket\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.948995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948923 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-kube-api-access-648tx\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-kube-api-access-648tx\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.948995 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948951 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-certs\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.949110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.948994 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istiod-ca-cert\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.949110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949067 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-data\") pod \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\" (UID: \"2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7\") " Oct 30 22:52:22.949239 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949124 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-credential-socket" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:22.949392 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949370 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-certs" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:22.949489 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949468 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istiod-ca-cert" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:22.949547 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949283 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-socket" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:22.949612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949577 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-data" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:22.949727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949708 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:22.949786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949730 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:22.949786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949739 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:22.949786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949747 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:22.949786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.949756 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:22.951093 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.951069 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-envoy" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:22.951226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.951200 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-kube-api-access-648tx" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "kube-api-access-648tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:22.951315 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.951296 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-podinfo" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:52:22.952240 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.952212 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-token" pod "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" (UID: "2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:22.977316 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977296 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9"] Oct 30 22:52:22.977889 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977875 4991 cpu_manager.go:414] "containerMap: removing container" podUID="69284587-4eeb-462e-938c-e3b7f3bb85fb" containerName="maas-api" Oct 30 22:52:22.977945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977890 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69284587-4eeb-462e-938c-e3b7f3bb85fb" containerName="maas-api" Oct 30 22:52:22.977945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977909 4991 cpu_manager.go:414] "containerMap: removing container" podUID="dbfd066a-4b0b-4929-904f-aaf4d73d9dab" containerName="maas-api" Oct 30 22:52:22.977945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977914 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfd066a-4b0b-4929-904f-aaf4d73d9dab" containerName="maas-api" Oct 30 22:52:22.977945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977921 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" Oct 30 22:52:22.977945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977926 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" Oct 30 22:52:22.977945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977934 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f946456d-2550-4e3f-adce-ff9fe63789cb" containerName="maas-api" Oct 30 22:52:22.977945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977940 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f946456d-2550-4e3f-adce-ff9fe63789cb" containerName="maas-api" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977952 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977957 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977967 4991 cpu_manager.go:414] "containerMap: removing container" podUID="895dfcd2-d9dd-4120-98e9-12f581eee595" containerName="maas-api" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977971 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895dfcd2-d9dd-4120-98e9-12f581eee595" containerName="maas-api" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977980 4991 cpu_manager.go:414] "containerMap: removing container" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977985 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977994 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.977999 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978005 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978009 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978017 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978021 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978036 4991 cpu_manager.go:414] "containerMap: removing container" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978040 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978049 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978054 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978062 4991 cpu_manager.go:414] "containerMap: removing container" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978066 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978143 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978152 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978159 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="69284587-4eeb-462e-938c-e3b7f3bb85fb" containerName="maas-api" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978167 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfd066a-4b0b-4929-904f-aaf4d73d9dab" containerName="maas-api" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978172 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978178 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f946456d-2550-4e3f-adce-ff9fe63789cb" containerName="maas-api" Oct 30 22:52:22.978177 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978185 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="768d126b-8220-489e-91df-595b1bdce9b6" containerName="istio-proxy" Oct 30 22:52:22.979103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978192 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" Oct 30 22:52:22.979103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978198 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="895dfcd2-d9dd-4120-98e9-12f581eee595" containerName="maas-api" Oct 30 22:52:22.979103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978204 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7669c06-6d57-4547-b94b-9e5f07a55a36" containerName="istio-proxy" Oct 30 22:52:22.979103 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.978209 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45e44a5-3029-43ff-a870-4190e9a041ec" containerName="main" Oct 30 22:52:22.982943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:22.982927 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.052038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.052001 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648tx\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-kube-api-access-648tx\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:23.052038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.052040 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:23.052281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.052055 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:23.052281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.052069 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:23.082366 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.082262 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9"] Oct 30 22:52:23.103154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.103128 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3b871a-openshift-default-dockercfg-s9764\"" Oct 30 22:52:23.152827 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.152788 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-socket\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.152852 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-token\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.152942 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77b4f03-6347-4591-919d-0aa3ba2453fc-istiod-ca-cert\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153033 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.153000 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-certs\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153174 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.153133 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-envoy\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153216 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.153174 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-podinfo\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153350 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.153305 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-credential-socket\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.153390 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hgj\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-kube-api-access-66hgj\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.153480 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.153463 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-data\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.254728 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.254695 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-envoy\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.254988 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.254962 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-podinfo\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.255745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.255722 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-credential-socket\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.259575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.259553 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hgj\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-kube-api-access-66hgj\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.260110 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.259369 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-podinfo\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.260237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.256111 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-credential-socket\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.260418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.260397 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-data\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.260584 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.260568 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-socket\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.260768 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.260743 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-token\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.260921 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.260903 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77b4f03-6347-4591-919d-0aa3ba2453fc-istiod-ca-cert\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.261039 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.261014 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-socket\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.261438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.261125 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-certs\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.261438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.260773 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-data\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.261725 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.261702 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-envoy\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.261868 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.261839 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-certs\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.264638 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.264610 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77b4f03-6347-4591-919d-0aa3ba2453fc-istiod-ca-cert\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.317864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.317830 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c"] Oct 30 22:52:23.321782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.321760 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f39eb7-openshift-default-8f6d9df55-5cl9c"] Oct 30 22:52:23.322176 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.322153 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hgj\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-kube-api-access-66hgj\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.323007 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.322983 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-token\") pod \"e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.597442 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.597416 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:23.877181 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:23.877105 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" path="/var/lib/kubelet/pods/2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7/volumes" Oct 30 22:52:24.036569 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.036527 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9"] Oct 30 22:52:24.037585 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:52:24.037558 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77b4f03_6347_4591_919d_0aa3ba2453fc.slice/crio-be1a6088258b0159f9fe8604c62e28a6e11aecea9c6fd91232abd463203eaf4f WatchSource:0}: Error finding container be1a6088258b0159f9fe8604c62e28a6e11aecea9c6fd91232abd463203eaf4f: Status 404 returned error can't find the container with id be1a6088258b0159f9fe8604c62e28a6e11aecea9c6fd91232abd463203eaf4f Oct 30 22:52:24.040707 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.040667 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:52:24.040815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.040763 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:52:24.040815 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.040808 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:52:24.670250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.670194 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:52:24.779265 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779236 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-data\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779297 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-kube-api-access-knj2n\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-kube-api-access-knj2n\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779345 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-credential-socket\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779396 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-token\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-token\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779429 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-socket\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779464 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istiod-ca-cert\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779523 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-data" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.779670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779536 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-podinfo\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779670 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779617 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-credential-socket" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.779883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779725 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-socket" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.779883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779744 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-certs\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.779883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779786 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-envoy\") pod \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\" (UID: \"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64\") " Oct 30 22:52:24.780020 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.779898 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istiod-ca-cert" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:24.780020 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.780007 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-certs" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.780384 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.780357 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.780530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.780387 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.780530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.780403 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.780530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.780418 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.780530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.780431 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.781725 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.781701 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-podinfo" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:52:24.781725 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.781712 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-token" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:24.781876 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.781852 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-envoy" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.782055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.782017 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-kube-api-access-knj2n" pod "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" (UID: "f8157601-ca44-4ce8-9a43-1bcf4f7dfa64"). InnerVolumeSpecName "kube-api-access-knj2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:24.882364 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.882315 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.882737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.882367 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knj2n\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-kube-api-access-knj2n\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.882737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.882385 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.882737 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.882399 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.898825 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.898792 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" event={"ID":"e77b4f03-6347-4591-919d-0aa3ba2453fc","Type":"ContainerStarted","Data":"1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7"} Oct 30 22:52:24.898941 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.898850 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" event={"ID":"e77b4f03-6347-4591-919d-0aa3ba2453fc","Type":"ContainerStarted","Data":"be1a6088258b0159f9fe8604c62e28a6e11aecea9c6fd91232abd463203eaf4f"} Oct 30 22:52:24.900760 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.900733 4991 generic.go:356] "Generic (PLEG): container finished" podID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerID="7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8" exitCode=0 Oct 30 22:52:24.900848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.900783 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" Oct 30 22:52:24.900848 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.900815 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" event={"ID":"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64","Type":"ContainerDied","Data":"7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8"} Oct 30 22:52:24.900934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.900864 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp" event={"ID":"f8157601-ca44-4ce8-9a43-1bcf4f7dfa64","Type":"ContainerDied","Data":"439bf4d6f9f95faae68aba8f1ae85d4c6d0afb2cd4567e8fc567ae83b8cd2546"} Oct 30 22:52:24.900934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.900887 4991 scope.go:122] "RemoveContainer" containerID="7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8" Oct 30 22:52:24.911487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.911470 4991 scope.go:122] "RemoveContainer" containerID="7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4" Oct 30 22:52:24.920244 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.920224 4991 scope.go:122] "RemoveContainer" containerID="7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8" Oct 30 22:52:24.920615 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:24.920575 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8\": container with ID starting with 7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8 not found: ID does not exist" containerID="7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8" Oct 30 22:52:24.920615 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.920602 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8"} err="failed to get container status \"7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8\": rpc error: code = NotFound desc = could not find container \"7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8\": container with ID starting with 7ea7d1250cc14dd5aa345e39d04684bf628d92ceacb165ccea9f86cd522f34e8 not found: ID does not exist" Oct 30 22:52:24.920730 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.920619 4991 scope.go:122] "RemoveContainer" containerID="7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4" Oct 30 22:52:24.920910 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:24.920887 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4\": container with ID starting with 7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4 not found: ID does not exist" containerID="7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4" Oct 30 22:52:24.920961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:24.920919 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4"} err="failed to get container status \"7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4\": rpc error: code = NotFound desc = could not find container \"7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4\": container with ID starting with 7c0f41e529e430f6a0cf793bea21eba144d037749370b5e4c5295c553cba2cc4 not found: ID does not exist" Oct 30 22:52:25.023676 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.023637 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podStartSLOduration=3.023625932 podStartE2EDuration="3.023625932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:25.020309106 +0000 UTC m=+3965.319718985" watchObservedRunningTime="2026-10-30 22:52:25.023625932 +0000 UTC m=+3965.323035814" Oct 30 22:52:25.155872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.155836 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp"] Oct 30 22:52:25.159987 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.159957 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-bs8hp"] Oct 30 22:52:25.598273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.598237 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:25.603932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.603908 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:25.881549 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.881472 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" path="/var/lib/kubelet/pods/f8157601-ca44-4ce8-9a43-1bcf4f7dfa64/volumes" Oct 30 22:52:25.909620 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.909594 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:25.910593 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:25.910573 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:52:34.330887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.330850 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf"] Oct 30 22:52:34.331424 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331410 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" Oct 30 22:52:34.331486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331424 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" Oct 30 22:52:34.331486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331441 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" Oct 30 22:52:34.331486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331447 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" Oct 30 22:52:34.331486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331454 4991 cpu_manager.go:414] "containerMap: removing container" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" Oct 30 22:52:34.331486 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331460 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" Oct 30 22:52:34.331665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331563 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1440bd-b906-43d9-9a0b-9a2ed036abe7" containerName="istio-proxy" Oct 30 22:52:34.331665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331572 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6571ec-fd75-4e3c-ad5b-1337d6cfefb7" containerName="istio-proxy" Oct 30 22:52:34.331665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331583 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" Oct 30 22:52:34.331665 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.331592 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad2290c-3dae-4d9a-b985-8820a0611b5c" containerName="istio-proxy" Oct 30 22:52:34.335297 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.335278 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:34.387132 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.387107 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5f059b-0888-4996-9ab2-45bf7606cd44-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:34.387362 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.387300 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8z2f\" (UniqueName: \"kubernetes.io/projected/8a5f059b-0888-4996-9ab2-45bf7606cd44-kube-api-access-l8z2f\") pod \"payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:34.395065 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.395038 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf"] Oct 30 22:52:34.463347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.461736 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3b871a\"" Oct 30 22:52:34.463347 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.461875 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3b871a-dockercfg-qhq5x\"" Oct 30 22:52:34.494363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.489581 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5f059b-0888-4996-9ab2-45bf7606cd44-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:34.494363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.490676 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8z2f\" (UniqueName: \"kubernetes.io/projected/8a5f059b-0888-4996-9ab2-45bf7606cd44-kube-api-access-l8z2f\") pod \"payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:34.494363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.491881 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5f059b-0888-4996-9ab2-45bf7606cd44-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:34.553003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.552969 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8z2f\" (UniqueName: \"kubernetes.io/projected/8a5f059b-0888-4996-9ab2-45bf7606cd44-kube-api-access-l8z2f\") pod \"payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:34.648948 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:34.648870 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:35.031830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.031809 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf"] Oct 30 22:52:35.034434 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:52:35.034406 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5f059b_0888_4996_9ab2_45bf7606cd44.slice/crio-8210a00ad7c8a0502b2baf6e18e7120ca78d5eb8e725ba7fa6a6ba45033ceed8 WatchSource:0}: Error finding container 8210a00ad7c8a0502b2baf6e18e7120ca78d5eb8e725ba7fa6a6ba45033ceed8: Status 404 returned error can't find the container with id 8210a00ad7c8a0502b2baf6e18e7120ca78d5eb8e725ba7fa6a6ba45033ceed8 Oct 30 22:52:35.103794 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.103764 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv"] Oct 30 22:52:35.104044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.103996 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" podUID="0f395d08-cdec-44fe-84ba-56084638c996" containerName="maas-api" containerID="cri-o://4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883" gracePeriod=30 Oct 30 22:52:35.383240 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.383213 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:52:35.523955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.523916 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-metrics-certs\") pod \"0f395d08-cdec-44fe-84ba-56084638c996\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " Oct 30 22:52:35.524130 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.523979 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-maas-api-tls\") pod \"0f395d08-cdec-44fe-84ba-56084638c996\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " Oct 30 22:52:35.524130 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.524031 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f395d08-cdec-44fe-84ba-56084638c996-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f395d08-cdec-44fe-84ba-56084638c996-openshift-service-ca\") pod \"0f395d08-cdec-44fe-84ba-56084638c996\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " Oct 30 22:52:35.524277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.524237 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f395d08-cdec-44fe-84ba-56084638c996-kube-api-access-cmsbr\" (UniqueName: \"kubernetes.io/projected/0f395d08-cdec-44fe-84ba-56084638c996-kube-api-access-cmsbr\") pod \"0f395d08-cdec-44fe-84ba-56084638c996\" (UID: \"0f395d08-cdec-44fe-84ba-56084638c996\") " Oct 30 22:52:35.524487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.524455 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f395d08-cdec-44fe-84ba-56084638c996-openshift-service-ca" pod "0f395d08-cdec-44fe-84ba-56084638c996" (UID: "0f395d08-cdec-44fe-84ba-56084638c996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:35.524698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.524681 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f395d08-cdec-44fe-84ba-56084638c996-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:35.526674 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.526644 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-metrics-certs" pod "0f395d08-cdec-44fe-84ba-56084638c996" (UID: "0f395d08-cdec-44fe-84ba-56084638c996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:35.526804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.526654 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-maas-api-tls" pod "0f395d08-cdec-44fe-84ba-56084638c996" (UID: "0f395d08-cdec-44fe-84ba-56084638c996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:35.526804 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.526786 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f395d08-cdec-44fe-84ba-56084638c996-kube-api-access-cmsbr" pod "0f395d08-cdec-44fe-84ba-56084638c996" (UID: "0f395d08-cdec-44fe-84ba-56084638c996"). InnerVolumeSpecName "kube-api-access-cmsbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:35.625631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.625605 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmsbr\" (UniqueName: \"kubernetes.io/projected/0f395d08-cdec-44fe-84ba-56084638c996-kube-api-access-cmsbr\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:35.625631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.625631 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:35.625780 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:35.625643 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f395d08-cdec-44fe-84ba-56084638c996-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:36.007344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.006554 4991 generic.go:356] "Generic (PLEG): container finished" podID="0f395d08-cdec-44fe-84ba-56084638c996" containerID="4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883" exitCode=0 Oct 30 22:52:36.007344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.006818 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" event={"ID":"0f395d08-cdec-44fe-84ba-56084638c996","Type":"ContainerDied","Data":"4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883"} Oct 30 22:52:36.007344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.006875 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" event={"ID":"0f395d08-cdec-44fe-84ba-56084638c996","Type":"ContainerDied","Data":"7519e0c916a85f08cc954c396f8456921c4e0616f9d32d8b5475b9c75742b96c"} Oct 30 22:52:36.007344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.006905 4991 scope.go:122] "RemoveContainer" containerID="4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883" Oct 30 22:52:36.007344 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.007062 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv" Oct 30 22:52:36.024109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.024078 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" event={"ID":"8a5f059b-0888-4996-9ab2-45bf7606cd44","Type":"ContainerStarted","Data":"cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544"} Oct 30 22:52:36.024285 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.024135 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" event={"ID":"8a5f059b-0888-4996-9ab2-45bf7606cd44","Type":"ContainerStarted","Data":"8210a00ad7c8a0502b2baf6e18e7120ca78d5eb8e725ba7fa6a6ba45033ceed8"} Oct 30 22:52:36.024926 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.024902 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:36.044194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.044090 4991 scope.go:122] "RemoveContainer" containerID="4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883" Oct 30 22:52:36.044612 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:36.044571 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883\": container with ID starting with 4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883 not found: ID does not exist" containerID="4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883" Oct 30 22:52:36.044720 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.044606 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883"} err="failed to get container status \"4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883\": rpc error: code = NotFound desc = could not find container \"4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883\": container with ID starting with 4888a9f14a4aaacb3c5da587df1f107f68c8c727464a12ed42b37012e1143883 not found: ID does not exist" Oct 30 22:52:36.183756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.183653 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" podStartSLOduration=2.18363384 podStartE2EDuration="2.18363384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:36.178542678 +0000 UTC m=+3976.477952558" watchObservedRunningTime="2026-10-30 22:52:36.18363384 +0000 UTC m=+3976.483043719" Oct 30 22:52:36.306599 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.306556 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv"] Oct 30 22:52:36.310418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:36.310388 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-293c80-7c7454dbf8-xj6cv"] Oct 30 22:52:37.875981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:37.875949 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f395d08-cdec-44fe-84ba-56084638c996" path="/var/lib/kubelet/pods/0f395d08-cdec-44fe-84ba-56084638c996/volumes" Oct 30 22:52:44.379255 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:44.379224 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr"] Oct 30 22:52:44.379824 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:44.379442 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" containerID="cri-o://ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80" gracePeriod=30 Oct 30 22:52:47.911903 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:47.911862 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:52:48.041049 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:48.041019 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:52:49.655515 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.655462 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:52:49.715178 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715154 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-podinfo\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715292 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715219 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-socket\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715292 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715262 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-credential-socket\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715292 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715285 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c032c362-3944-42cf-8576-2c74c8f9dca2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032c362-3944-42cf-8576-2c74c8f9dca2-istiod-ca-cert\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715347 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-kube-api-access-2shgw\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-kube-api-access-2shgw\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715371 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-data\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715397 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-token\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-token\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715441 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-envoy\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715485 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-certs\") pod \"c032c362-3944-42cf-8576-2c74c8f9dca2\" (UID: \"c032c362-3944-42cf-8576-2c74c8f9dca2\") " Oct 30 22:52:49.715714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715592 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-socket" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:49.715714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715612 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-credential-socket" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:49.715899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715833 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-certs" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:49.715899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715885 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-data" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:49.716018 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.715906 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c032c362-3944-42cf-8576-2c74c8f9dca2-istiod-ca-cert" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:49.716078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.716036 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.716078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.716056 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.716078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.716073 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032c362-3944-42cf-8576-2c74c8f9dca2-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.716339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.716088 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.716339 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.716101 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.717660 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.717616 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-podinfo" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:52:49.717771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.717677 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-envoy" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:49.717771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.717688 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-token" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:49.717771 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.717752 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-kube-api-access-2shgw" pod "c032c362-3944-42cf-8576-2c74c8f9dca2" (UID: "c032c362-3944-42cf-8576-2c74c8f9dca2"). InnerVolumeSpecName "kube-api-access-2shgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:49.816963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.816933 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.816963 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.816961 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2shgw\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-kube-api-access-2shgw\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.817116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.816978 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:49.817116 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:49.816990 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032c362-3944-42cf-8576-2c74c8f9dca2-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:50.157710 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.157674 4991 generic.go:356] "Generic (PLEG): container finished" podID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerID="ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80" exitCode=0 Oct 30 22:52:50.157890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.157753 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" Oct 30 22:52:50.157890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.157770 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" event={"ID":"c032c362-3944-42cf-8576-2c74c8f9dca2","Type":"ContainerDied","Data":"ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80"} Oct 30 22:52:50.157890 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.157877 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr" event={"ID":"c032c362-3944-42cf-8576-2c74c8f9dca2","Type":"ContainerDied","Data":"f00d31a1453f2ff4cc1e377a121cee3d56320e9c7db2b4a56e3b67ea700bac02"} Oct 30 22:52:50.158009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.157907 4991 scope.go:122] "RemoveContainer" containerID="ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80" Oct 30 22:52:50.168085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.168063 4991 scope.go:122] "RemoveContainer" containerID="553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49" Oct 30 22:52:50.178036 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.178011 4991 scope.go:122] "RemoveContainer" containerID="ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80" Oct 30 22:52:50.178440 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:50.178315 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80\": container with ID starting with ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80 not found: ID does not exist" containerID="ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80" Oct 30 22:52:50.178546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.178437 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80"} err="failed to get container status \"ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80\": rpc error: code = NotFound desc = could not find container \"ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80\": container with ID starting with ab1d09dce1c1fd969a87ef8317dd1688a87d3ca56e8b8c05bb370ac2103dfd80 not found: ID does not exist" Oct 30 22:52:50.178546 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.178470 4991 scope.go:122] "RemoveContainer" containerID="553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49" Oct 30 22:52:50.179199 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:50.179172 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49\": container with ID starting with 553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49 not found: ID does not exist" containerID="553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49" Oct 30 22:52:50.179350 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.179303 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49"} err="failed to get container status \"553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49\": rpc error: code = NotFound desc = could not find container \"553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49\": container with ID starting with 553e0ec73adba4853c84b9857f4c41d299059090878bf2db8d60a3f05e92fa49 not found: ID does not exist" Oct 30 22:52:50.253970 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.253940 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq"] Oct 30 22:52:50.254225 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.254200 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" containerID="cri-o://5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75" gracePeriod=30 Oct 30 22:52:50.333595 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.333509 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr"] Oct 30 22:52:50.342605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:50.342574 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-293c80-openshift-default-849b7998bf-nntwr"] Oct 30 22:52:51.874616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:51.874582 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" path="/var/lib/kubelet/pods/c032c362-3944-42cf-8576-2c74c8f9dca2/volumes" Oct 30 22:52:52.482521 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:52.482485 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:52:52.482755 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:52.482557 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:52:55.523485 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.523422 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:52:55.598808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.598777 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-token\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-token\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.598957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.598832 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-certs\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.598957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.598930 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-credential-socket\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.599038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.598964 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istiod-ca-cert\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.599038 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.598999 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-data\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.599135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599114 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-certs" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:55.599192 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599125 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-envoy\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.599192 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599179 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-podinfo\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.599298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599220 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-socket\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.599298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599200 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-credential-socket" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:55.599298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599245 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-data" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:55.599298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599285 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-kube-api-access-7kj5f\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-kube-api-access-7kj5f\") pod \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\" (UID: \"f3b23fab-1eba-43bf-ae70-bff4daba1fcc\") " Oct 30 22:52:55.599298 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599288 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istiod-ca-cert" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:55.599612 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599532 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-socket" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:55.599820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599795 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.599820 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599821 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.600015 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599835 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.600015 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599849 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.600015 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.599862 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.601388 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.601366 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-podinfo" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:52:55.601467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.601399 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-kube-api-access-7kj5f" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "kube-api-access-7kj5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:55.601585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.601568 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-envoy" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:55.601711 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.601688 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-token" pod "f3b23fab-1eba-43bf-ae70-bff4daba1fcc" (UID: "f3b23fab-1eba-43bf-ae70-bff4daba1fcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:55.701016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.700987 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kj5f\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-kube-api-access-7kj5f\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.701016 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.701013 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.701179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.701024 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.701179 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:55.701033 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b23fab-1eba-43bf-ae70-bff4daba1fcc-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:56.201419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.201307 4991 generic.go:356] "Generic (PLEG): container finished" podID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerID="5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75" exitCode=0 Oct 30 22:52:56.201419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.201354 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" event={"ID":"f3b23fab-1eba-43bf-ae70-bff4daba1fcc","Type":"ContainerDied","Data":"5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75"} Oct 30 22:52:56.201419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.201394 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" Oct 30 22:52:56.201419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.201413 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq" event={"ID":"f3b23fab-1eba-43bf-ae70-bff4daba1fcc","Type":"ContainerDied","Data":"10cb31ebbdfc2e8705c0cea67bc1d42a261981e734861fb66bcf4f56c8672e74"} Oct 30 22:52:56.201699 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.201436 4991 scope.go:122] "RemoveContainer" containerID="5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75" Oct 30 22:52:56.211700 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.211682 4991 scope.go:122] "RemoveContainer" containerID="0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30" Oct 30 22:52:56.220301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.220284 4991 scope.go:122] "RemoveContainer" containerID="5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75" Oct 30 22:52:56.220581 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:56.220562 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75\": container with ID starting with 5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75 not found: ID does not exist" containerID="5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75" Oct 30 22:52:56.220654 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.220590 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75"} err="failed to get container status \"5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75\": rpc error: code = NotFound desc = could not find container \"5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75\": container with ID starting with 5ee211abdf5d8b6c76165e62ee333aa8d801b7cead4bb435cdcd5a76b86d5f75 not found: ID does not exist" Oct 30 22:52:56.220654 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.220608 4991 scope.go:122] "RemoveContainer" containerID="0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30" Oct 30 22:52:56.220872 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:52:56.220852 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30\": container with ID starting with 0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30 not found: ID does not exist" containerID="0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30" Oct 30 22:52:56.220915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.220879 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30"} err="failed to get container status \"0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30\": rpc error: code = NotFound desc = could not find container \"0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30\": container with ID starting with 0916434c983de30ab7293b805b7eda3d0aa72e785a8fde517b6a55888048ec30 not found: ID does not exist" Oct 30 22:52:56.373544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.373517 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq"] Oct 30 22:52:56.423870 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:56.423844 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-37e79b-openshift-default-57f4bf6db7-2bxhq"] Oct 30 22:52:57.894381 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:52:57.891957 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" path="/var/lib/kubelet/pods/f3b23fab-1eba-43bf-ae70-bff4daba1fcc/volumes" Oct 30 22:53:08.768559 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:08.768475 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2"] Oct 30 22:53:08.768951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:08.768759 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" containerID="cri-o://70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64" gracePeriod=30 Oct 30 22:53:13.791603 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:13.791574 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 22:53:13.791899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:13.791623 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 22:53:14.026447 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.026424 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:53:14.116259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116172 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-socket\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116277 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-kube-api-access-8mfzk\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-kube-api-access-8mfzk\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116344 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-certs\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116448 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116379 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-data\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116451 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-envoy\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116479 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-token\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116532 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-podinfo\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116565 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-credential-socket\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.116636 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116619 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istiod-ca-cert\") pod \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\" (UID: \"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6\") " Oct 30 22:53:14.117999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.116961 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-data" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:14.117999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.117261 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.117999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.117508 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-certs" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:14.117999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.117639 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-credential-socket" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:14.117999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.117660 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istiod-ca-cert" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:14.117999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.117955 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-socket" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:14.119897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.119867 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-kube-api-access-8mfzk" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "kube-api-access-8mfzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:14.120001 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.119974 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-podinfo" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:53:14.120529 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.120502 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-token" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:14.120634 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.120580 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-envoy" pod "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" (UID: "ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:14.218150 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218115 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.218334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218157 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.218334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218176 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfzk\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-kube-api-access-8mfzk\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.218334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218191 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.218334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218205 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.218334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218219 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.218334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218235 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.218334 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.218249 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:14.334616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.334579 4991 generic.go:356] "Generic (PLEG): container finished" podID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerID="70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64" exitCode=0 Oct 30 22:53:14.334783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.334641 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" Oct 30 22:53:14.334783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.334670 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" event={"ID":"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6","Type":"ContainerDied","Data":"70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64"} Oct 30 22:53:14.334783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.334761 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2" event={"ID":"ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6","Type":"ContainerDied","Data":"573db9f45cd5ffae326e085c36e6808d0864f7430ea8484ae382a351abd1c9f3"} Oct 30 22:53:14.334943 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.334790 4991 scope.go:122] "RemoveContainer" containerID="70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64" Oct 30 22:53:14.345139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.345123 4991 scope.go:122] "RemoveContainer" containerID="69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54" Oct 30 22:53:14.354017 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.354001 4991 scope.go:122] "RemoveContainer" containerID="70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64" Oct 30 22:53:14.354331 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:53:14.354299 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64\": container with ID starting with 70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64 not found: ID does not exist" containerID="70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64" Oct 30 22:53:14.354420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.354342 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64"} err="failed to get container status \"70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64\": rpc error: code = NotFound desc = could not find container \"70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64\": container with ID starting with 70de0ed7575b1f3818737ce6aab8a775793a7cc7d22d3df4d3e6f75fa667bc64 not found: ID does not exist" Oct 30 22:53:14.354420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.354360 4991 scope.go:122] "RemoveContainer" containerID="69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54" Oct 30 22:53:14.354665 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:53:14.354648 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54\": container with ID starting with 69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54 not found: ID does not exist" containerID="69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54" Oct 30 22:53:14.354715 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.354670 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54"} err="failed to get container status \"69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54\": rpc error: code = NotFound desc = could not find container \"69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54\": container with ID starting with 69ae5715b13d4b27303084c271abdb51c2d615388a1e2fca101f78502bceca54 not found: ID does not exist" Oct 30 22:53:14.504439 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.504416 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2"] Oct 30 22:53:14.508684 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:14.508666 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fc6466-openshift-default-6dcbd54cd7-f8vm2"] Oct 30 22:53:15.875581 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:15.875546 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" path="/var/lib/kubelet/pods/ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6/volumes" Oct 30 22:53:17.911595 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:53:17.911560 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:53:33.100880 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.100846 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6"] Oct 30 22:53:33.101401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101355 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" Oct 30 22:53:33.101401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101373 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" Oct 30 22:53:33.101401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101389 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" Oct 30 22:53:33.101401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101398 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101413 4991 cpu_manager.go:414] "containerMap: removing container" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101419 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101430 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101435 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101454 4991 cpu_manager.go:414] "containerMap: removing container" podUID="0f395d08-cdec-44fe-84ba-56084638c996" containerName="maas-api" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101458 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f395d08-cdec-44fe-84ba-56084638c996" containerName="maas-api" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101467 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101471 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101481 4991 cpu_manager.go:414] "containerMap: removing container" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" Oct 30 22:53:33.101550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101485 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" Oct 30 22:53:33.101864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101575 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f395d08-cdec-44fe-84ba-56084638c996" containerName="maas-api" Oct 30 22:53:33.101864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101586 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" Oct 30 22:53:33.101864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101593 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" Oct 30 22:53:33.101864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101600 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8157601-ca44-4ce8-9a43-1bcf4f7dfa64" containerName="istio-proxy" Oct 30 22:53:33.101864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101607 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" Oct 30 22:53:33.101864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.101613 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4e09b8-0c01-42ab-a5b7-ff487d4c0aa6" containerName="istio-proxy" Oct 30 22:53:33.105028 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.105009 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.166757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.166724 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3b871a-serving-cert\"" Oct 30 22:53:33.217092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.217061 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-metrics-certs\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.217266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.217110 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g627\" (UniqueName: \"kubernetes.io/projected/66834ead-f417-4228-bd5f-a5cde6ca346f-kube-api-access-7g627\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.217266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.217202 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66834ead-f417-4228-bd5f-a5cde6ca346f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.217266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.217229 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.217907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.217882 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6"] Oct 30 22:53:33.318966 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.318923 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-metrics-certs\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.319243 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.319218 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g627\" (UniqueName: \"kubernetes.io/projected/66834ead-f417-4228-bd5f-a5cde6ca346f-kube-api-access-7g627\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.319409 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.319390 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66834ead-f417-4228-bd5f-a5cde6ca346f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.319503 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.319437 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.319726 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:53:33.319702 4991 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-serving-cert: secret "maas-api-e2e-extmod-b-3b871a-serving-cert" not found Oct 30 22:53:33.319822 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:53:33.319777 4991 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls podName:66834ead-f417-4228-bd5f-a5cde6ca346f nodeName:}" failed. No retries permitted until 2026-10-30 22:53:33.819754756 +0000 UTC m=+4034.119164616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls") pod "maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" (UID: "66834ead-f417-4228-bd5f-a5cde6ca346f") : secret "maas-api-e2e-extmod-b-3b871a-serving-cert" not found Oct 30 22:53:33.320594 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.320566 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66834ead-f417-4228-bd5f-a5cde6ca346f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.340261 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.340192 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-metrics-certs\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.382449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.382384 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g627\" (UniqueName: \"kubernetes.io/projected/66834ead-f417-4228-bd5f-a5cde6ca346f-kube-api-access-7g627\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.826467 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.826412 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:33.828997 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:33.828961 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:34.018608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:34.018575 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:34.406688 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:34.406550 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6"] Oct 30 22:53:34.412010 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:53:34.411977 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66834ead_f417_4228_bd5f_a5cde6ca346f.slice/crio-33c1ea3b9f5fff2c560813d4d4031a802a30506e58492550f2e7599346385608 WatchSource:0}: Error finding container 33c1ea3b9f5fff2c560813d4d4031a802a30506e58492550f2e7599346385608: Status 404 returned error can't find the container with id 33c1ea3b9f5fff2c560813d4d4031a802a30506e58492550f2e7599346385608 Oct 30 22:53:34.413437 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:34.413420 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:53:34.500661 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:34.500624 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" event={"ID":"66834ead-f417-4228-bd5f-a5cde6ca346f","Type":"ContainerStarted","Data":"33c1ea3b9f5fff2c560813d4d4031a802a30506e58492550f2e7599346385608"} Oct 30 22:53:35.509107 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:35.509071 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" event={"ID":"66834ead-f417-4228-bd5f-a5cde6ca346f","Type":"ContainerStarted","Data":"b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c"} Oct 30 22:53:35.509466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:35.509159 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:35.623173 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:35.623116 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" podStartSLOduration=1.6472142760000001 podStartE2EDuration="2.623102855s" totalImagesPullingTime="975.888579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:35.621774008 +0000 UTC m=+4035.921183899" watchObservedRunningTime="2026-10-30 22:53:35.623102855 +0000 UTC m=+4035.922512734" Oct 30 22:53:40.402031 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:40.401995 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls"] Oct 30 22:53:40.402630 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:40.402189 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" containerID="cri-o://4501e2c7ebbdf35d8d158f8562169e90a80455912775b44d1327969831a8e792" gracePeriod=30 Oct 30 22:53:41.521135 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:41.521104 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:53:44.070979 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:44.070940 4991 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:53:44.071380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:44.070994 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:53:45.572475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.572431 4991 generic.go:356] "Generic (PLEG): container finished" podID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerID="4501e2c7ebbdf35d8d158f8562169e90a80455912775b44d1327969831a8e792" exitCode=0 Oct 30 22:53:45.572856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.572560 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" event={"ID":"c0ab5024-b7a9-496b-951d-ea388be139c5","Type":"ContainerDied","Data":"4501e2c7ebbdf35d8d158f8562169e90a80455912775b44d1327969831a8e792"} Oct 30 22:53:45.673899 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.673838 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:53:45.864025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.863989 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-socket\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864026 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-envoy\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864069 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-credential-socket\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864094 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-token\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-token\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864114 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ab5024-b7a9-496b-951d-ea388be139c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ab5024-b7a9-496b-951d-ea388be139c5-istiod-ca-cert\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864137 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-kube-api-access-gxxng\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-kube-api-access-gxxng\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864280 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864206 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-data\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864285 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-certs\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864334 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-socket" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.864575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864341 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-podinfo\") pod \"c0ab5024-b7a9-496b-951d-ea388be139c5\" (UID: \"c0ab5024-b7a9-496b-951d-ea388be139c5\") " Oct 30 22:53:45.864575 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864436 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-credential-socket" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.864894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864858 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-data" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.864894 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864871 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.865011 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.864969 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.865151 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.865127 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ab5024-b7a9-496b-951d-ea388be139c5-istiod-ca-cert" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:45.865237 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.865190 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-certs" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.866994 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.866967 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-podinfo" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:53:45.867128 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.867038 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-kube-api-access-gxxng" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "kube-api-access-gxxng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:45.867196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.867126 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-token" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:45.867723 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.867701 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-envoy" pod "c0ab5024-b7a9-496b-951d-ea388be139c5" (UID: "c0ab5024-b7a9-496b-951d-ea388be139c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.966408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.966346 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.966408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.966370 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.966408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.966380 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ab5024-b7a9-496b-951d-ea388be139c5-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.966408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.966389 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxxng\" (UniqueName: \"kubernetes.io/projected/c0ab5024-b7a9-496b-951d-ea388be139c5-kube-api-access-gxxng\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.966408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.966398 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.966408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.966406 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ab5024-b7a9-496b-951d-ea388be139c5-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.966408 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:45.966414 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ab5024-b7a9-496b-951d-ea388be139c5-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:46.587086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:46.587050 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" Oct 30 22:53:46.587086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:46.587068 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls" event={"ID":"c0ab5024-b7a9-496b-951d-ea388be139c5","Type":"ContainerDied","Data":"5b451679276fb293a4c0ff8ff508a37396ad84df1a7552ad9075e47c1504a7ef"} Oct 30 22:53:46.587642 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:46.587150 4991 scope.go:122] "RemoveContainer" containerID="4501e2c7ebbdf35d8d158f8562169e90a80455912775b44d1327969831a8e792" Oct 30 22:53:46.790132 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:46.790100 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls"] Oct 30 22:53:46.838250 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:46.838155 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-cllls"] Oct 30 22:53:47.874836 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:47.874805 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" path="/var/lib/kubelet/pods/c0ab5024-b7a9-496b-951d-ea388be139c5/volumes" Oct 30 22:53:47.909885 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:53:47.909850 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:53:56.951565 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:56.951519 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:53:56.952009 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:56.951602 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:53:59.704357 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:59.704303 4991 generic.go:356] "Generic (PLEG): container finished" podID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerID="1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7" exitCode=0 Oct 30 22:53:59.704875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:59.704393 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" event={"ID":"e77b4f03-6347-4591-919d-0aa3ba2453fc","Type":"ContainerDied","Data":"1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7"} Oct 30 22:53:59.704875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:59.704770 4991 scope.go:122] "RemoveContainer" containerID="1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7" Oct 30 22:53:59.706226 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:59.706185 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:53:59.706373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:59.706269 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:53:59.706373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:53:59.706345 4991 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:54:00.713288 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:00.713254 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" event={"ID":"e77b4f03-6347-4591-919d-0aa3ba2453fc","Type":"ContainerStarted","Data":"14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4"} Oct 30 22:54:01.182337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.182297 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9"] Oct 30 22:54:01.182538 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.182516 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" podUID="92343dce-b0c6-44f8-a913-e718e6eb0ad7" containerName="maas-api" containerID="cri-o://698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3" gracePeriod=30 Oct 30 22:54:01.486418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.486395 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:54:01.574860 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.574825 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92343dce-b0c6-44f8-a913-e718e6eb0ad7-kube-api-access-kvcpt\" (UniqueName: \"kubernetes.io/projected/92343dce-b0c6-44f8-a913-e718e6eb0ad7-kube-api-access-kvcpt\") pod \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " Oct 30 22:54:01.575052 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.574866 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls\") pod \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " Oct 30 22:54:01.575052 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.574998 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92343dce-b0c6-44f8-a913-e718e6eb0ad7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92343dce-b0c6-44f8-a913-e718e6eb0ad7-openshift-service-ca\") pod \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " Oct 30 22:54:01.575149 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.575058 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-metrics-certs\") pod \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\" (UID: \"92343dce-b0c6-44f8-a913-e718e6eb0ad7\") " Oct 30 22:54:01.575368 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.575344 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92343dce-b0c6-44f8-a913-e718e6eb0ad7-openshift-service-ca" pod "92343dce-b0c6-44f8-a913-e718e6eb0ad7" (UID: "92343dce-b0c6-44f8-a913-e718e6eb0ad7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:01.575631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.575596 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92343dce-b0c6-44f8-a913-e718e6eb0ad7-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:01.577074 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.577039 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls" pod "92343dce-b0c6-44f8-a913-e718e6eb0ad7" (UID: "92343dce-b0c6-44f8-a913-e718e6eb0ad7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:01.577074 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.577058 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-metrics-certs" pod "92343dce-b0c6-44f8-a913-e718e6eb0ad7" (UID: "92343dce-b0c6-44f8-a913-e718e6eb0ad7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:01.577220 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.577100 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92343dce-b0c6-44f8-a913-e718e6eb0ad7-kube-api-access-kvcpt" pod "92343dce-b0c6-44f8-a913-e718e6eb0ad7" (UID: "92343dce-b0c6-44f8-a913-e718e6eb0ad7"). InnerVolumeSpecName "kube-api-access-kvcpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:01.597872 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.597849 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:54:01.598080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.598059 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 22:54:01.598159 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.598110 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 22:54:01.677230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.677194 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcpt\" (UniqueName: \"kubernetes.io/projected/92343dce-b0c6-44f8-a913-e718e6eb0ad7-kube-api-access-kvcpt\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:01.677230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.677231 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:01.677457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.677245 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92343dce-b0c6-44f8-a913-e718e6eb0ad7-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:01.752976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.752937 4991 generic.go:356] "Generic (PLEG): container finished" podID="92343dce-b0c6-44f8-a913-e718e6eb0ad7" containerID="698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3" exitCode=0 Oct 30 22:54:01.753632 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.752985 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" Oct 30 22:54:01.753790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.752984 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" event={"ID":"92343dce-b0c6-44f8-a913-e718e6eb0ad7","Type":"ContainerDied","Data":"698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3"} Oct 30 22:54:01.753790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.753761 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9" event={"ID":"92343dce-b0c6-44f8-a913-e718e6eb0ad7","Type":"ContainerDied","Data":"52c66885bf3555f03b15865e295c3e66ecd7185d6e4af26ab821dd90340d252e"} Oct 30 22:54:01.753929 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.753814 4991 scope.go:122] "RemoveContainer" containerID="698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3" Oct 30 22:54:01.766210 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.766189 4991 scope.go:122] "RemoveContainer" containerID="698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3" Oct 30 22:54:01.766569 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:54:01.766550 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3\": container with ID starting with 698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3 not found: ID does not exist" containerID="698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3" Oct 30 22:54:01.766658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.766578 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3"} err="failed to get container status \"698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3\": rpc error: code = NotFound desc = could not find container \"698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3\": container with ID starting with 698f3fa82fbaa19fdb73c9a6f9c9773fee54340bac63cfa7727ea61439c216b3 not found: ID does not exist" Oct 30 22:54:01.910416 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.910384 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9"] Oct 30 22:54:01.914787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:01.914749 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-435923-7d9cbcd576-t76r9"] Oct 30 22:54:02.598217 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:02.598182 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 22:54:02.598432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:02.598235 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 22:54:03.598278 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:03.598245 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 22:54:03.598769 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:03.598300 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 22:54:03.874792 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:03.874712 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92343dce-b0c6-44f8-a913-e718e6eb0ad7" path="/var/lib/kubelet/pods/92343dce-b0c6-44f8-a913-e718e6eb0ad7/volumes" Oct 30 22:54:04.597999 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:04.597959 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 22:54:04.598185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:04.598018 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 22:54:05.598632 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:05.598598 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 22:54:05.599032 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:05.598650 4991 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 22:54:06.602041 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:06.602010 4991 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:54:06.602549 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:06.602533 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:54:06.603215 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:06.603193 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:54:17.913439 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:54:17.913399 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:54:30.806773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:30.806695 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7"] Oct 30 22:54:30.807307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:30.806907 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" podUID="cbacae8f-ac18-44ba-aedb-6ecb049209dc" containerName="maas-api" containerID="cri-o://3a8779b2b875412f56a7d0d20c5d85aaee64a16ac7e42ef9243c2ae312e447cb" gracePeriod=30 Oct 30 22:54:30.953918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:30.953885 4991 generic.go:356] "Generic (PLEG): container finished" podID="cbacae8f-ac18-44ba-aedb-6ecb049209dc" containerID="3a8779b2b875412f56a7d0d20c5d85aaee64a16ac7e42ef9243c2ae312e447cb" exitCode=0 Oct 30 22:54:30.954076 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:30.953964 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" event={"ID":"cbacae8f-ac18-44ba-aedb-6ecb049209dc","Type":"ContainerDied","Data":"3a8779b2b875412f56a7d0d20c5d85aaee64a16ac7e42ef9243c2ae312e447cb"} Oct 30 22:54:31.082196 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.082164 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:54:31.196652 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.196620 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls\") pod \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " Oct 30 22:54:31.196826 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.196675 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbacae8f-ac18-44ba-aedb-6ecb049209dc-kube-api-access-twgdb\" (UniqueName: \"kubernetes.io/projected/cbacae8f-ac18-44ba-aedb-6ecb049209dc-kube-api-access-twgdb\") pod \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " Oct 30 22:54:31.196911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.196826 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-metrics-certs\") pod \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " Oct 30 22:54:31.196911 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.196902 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbacae8f-ac18-44ba-aedb-6ecb049209dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbacae8f-ac18-44ba-aedb-6ecb049209dc-openshift-service-ca\") pod \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\" (UID: \"cbacae8f-ac18-44ba-aedb-6ecb049209dc\") " Oct 30 22:54:31.197292 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.197257 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbacae8f-ac18-44ba-aedb-6ecb049209dc-openshift-service-ca" pod "cbacae8f-ac18-44ba-aedb-6ecb049209dc" (UID: "cbacae8f-ac18-44ba-aedb-6ecb049209dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:31.197500 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.197389 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbacae8f-ac18-44ba-aedb-6ecb049209dc-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:31.199746 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.199718 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-metrics-certs" pod "cbacae8f-ac18-44ba-aedb-6ecb049209dc" (UID: "cbacae8f-ac18-44ba-aedb-6ecb049209dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:31.200072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.200014 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls" pod "cbacae8f-ac18-44ba-aedb-6ecb049209dc" (UID: "cbacae8f-ac18-44ba-aedb-6ecb049209dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:31.200946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.200921 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbacae8f-ac18-44ba-aedb-6ecb049209dc-kube-api-access-twgdb" pod "cbacae8f-ac18-44ba-aedb-6ecb049209dc" (UID: "cbacae8f-ac18-44ba-aedb-6ecb049209dc"). InnerVolumeSpecName "kube-api-access-twgdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:31.298790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.298759 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:31.298790 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.298789 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgdb\" (UniqueName: \"kubernetes.io/projected/cbacae8f-ac18-44ba-aedb-6ecb049209dc-kube-api-access-twgdb\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:31.298981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.298805 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbacae8f-ac18-44ba-aedb-6ecb049209dc-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:31.961425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.961388 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" Oct 30 22:54:31.961425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.961412 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7" event={"ID":"cbacae8f-ac18-44ba-aedb-6ecb049209dc","Type":"ContainerDied","Data":"571eda469aba42e870777494e93752ecaef761b8b7f5952fd67ee62a72134e17"} Oct 30 22:54:31.961931 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:31.961477 4991 scope.go:122] "RemoveContainer" containerID="3a8779b2b875412f56a7d0d20c5d85aaee64a16ac7e42ef9243c2ae312e447cb" Oct 30 22:54:32.117525 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:32.117494 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7"] Oct 30 22:54:32.121698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:32.121670 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5075-647df59bbc-qmwr7"] Oct 30 22:54:33.874712 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:33.874676 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbacae8f-ac18-44ba-aedb-6ecb049209dc" path="/var/lib/kubelet/pods/cbacae8f-ac18-44ba-aedb-6ecb049209dc/volumes" Oct 30 22:54:36.761580 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:36.761541 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75"] Oct 30 22:54:36.762162 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:36.761794 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" podUID="92d4fd17-1756-47a6-81df-e3d7174555f0" containerName="maas-api" containerID="cri-o://cf79fbdd97b89a0248e677dfa4cf7352cc282eb5a8d04fc8ebddaba404371a35" gracePeriod=30 Oct 30 22:54:36.994724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:36.994693 4991 generic.go:356] "Generic (PLEG): container finished" podID="92d4fd17-1756-47a6-81df-e3d7174555f0" containerID="cf79fbdd97b89a0248e677dfa4cf7352cc282eb5a8d04fc8ebddaba404371a35" exitCode=0 Oct 30 22:54:36.994887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:36.994768 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" event={"ID":"92d4fd17-1756-47a6-81df-e3d7174555f0","Type":"ContainerDied","Data":"cf79fbdd97b89a0248e677dfa4cf7352cc282eb5a8d04fc8ebddaba404371a35"} Oct 30 22:54:37.043242 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.043218 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:54:37.167034 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.166997 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d4fd17-1756-47a6-81df-e3d7174555f0-kube-api-access-nb6jr\" (UniqueName: \"kubernetes.io/projected/92d4fd17-1756-47a6-81df-e3d7174555f0-kube-api-access-nb6jr\") pod \"92d4fd17-1756-47a6-81df-e3d7174555f0\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " Oct 30 22:54:37.167207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.167049 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls\") pod \"92d4fd17-1756-47a6-81df-e3d7174555f0\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " Oct 30 22:54:37.167207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.167093 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d4fd17-1756-47a6-81df-e3d7174555f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d4fd17-1756-47a6-81df-e3d7174555f0-openshift-service-ca\") pod \"92d4fd17-1756-47a6-81df-e3d7174555f0\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " Oct 30 22:54:37.167207 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.167133 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-metrics-certs\") pod \"92d4fd17-1756-47a6-81df-e3d7174555f0\" (UID: \"92d4fd17-1756-47a6-81df-e3d7174555f0\") " Oct 30 22:54:37.167724 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.167691 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d4fd17-1756-47a6-81df-e3d7174555f0-openshift-service-ca" pod "92d4fd17-1756-47a6-81df-e3d7174555f0" (UID: "92d4fd17-1756-47a6-81df-e3d7174555f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:37.169840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.169808 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d4fd17-1756-47a6-81df-e3d7174555f0-kube-api-access-nb6jr" pod "92d4fd17-1756-47a6-81df-e3d7174555f0" (UID: "92d4fd17-1756-47a6-81df-e3d7174555f0"). InnerVolumeSpecName "kube-api-access-nb6jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:37.169977 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.169891 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls" pod "92d4fd17-1756-47a6-81df-e3d7174555f0" (UID: "92d4fd17-1756-47a6-81df-e3d7174555f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:37.170263 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.170244 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-metrics-certs" pod "92d4fd17-1756-47a6-81df-e3d7174555f0" (UID: "92d4fd17-1756-47a6-81df-e3d7174555f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:37.268857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.268762 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:37.268857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.268799 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6jr\" (UniqueName: \"kubernetes.io/projected/92d4fd17-1756-47a6-81df-e3d7174555f0-kube-api-access-nb6jr\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:37.268857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.268816 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92d4fd17-1756-47a6-81df-e3d7174555f0-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:37.268857 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:37.268830 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d4fd17-1756-47a6-81df-e3d7174555f0-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:38.010230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:38.010196 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" Oct 30 22:54:38.010678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:38.010231 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75" event={"ID":"92d4fd17-1756-47a6-81df-e3d7174555f0","Type":"ContainerDied","Data":"4c94502d9c91dc3233539b680c3de3c42a20f52eefafe194fcbeed4e74a9c66c"} Oct 30 22:54:38.010678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:38.010341 4991 scope.go:122] "RemoveContainer" containerID="cf79fbdd97b89a0248e677dfa4cf7352cc282eb5a8d04fc8ebddaba404371a35" Oct 30 22:54:38.202240 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:38.202164 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75"] Oct 30 22:54:38.248934 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:38.248896 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e524ce-8b9c66467-x5q75"] Oct 30 22:54:39.875662 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:39.875626 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d4fd17-1756-47a6-81df-e3d7174555f0" path="/var/lib/kubelet/pods/92d4fd17-1756-47a6-81df-e3d7174555f0/volumes" Oct 30 22:54:41.405897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:41.405859 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l"] Oct 30 22:54:41.406353 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:41.406072 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" containerID="cri-o://26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e" gracePeriod=30 Oct 30 22:54:41.972386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:41.972348 4991 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:54:41.972689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:41.972406 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:54:46.684643 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.684588 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:54:46.701672 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701645 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-data\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.701797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701706 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-kube-api-access-llx47\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-kube-api-access-llx47\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.701797 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701746 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae9d702-f807-4040-b5e8-e140bead935f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae9d702-f807-4040-b5e8-e140bead935f-istiod-ca-cert\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.701895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701837 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-envoy\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.701895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701841 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-data" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:46.701895 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701876 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-istio-token\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.702072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701924 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-certs\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.702072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.701981 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-socket\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.702072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.702066 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae9d702-f807-4040-b5e8-e140bead935f-istiod-ca-cert" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:46.702266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.702239 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-certs" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:46.702354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.702269 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-socket" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:46.702487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.702469 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae9d702-f807-4040-b5e8-e140bead935f-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.702544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.702491 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.702544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.702505 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.702544 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.702517 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.703896 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.703872 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-envoy" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:46.704264 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.704241 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-kube-api-access-llx47" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "kube-api-access-llx47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:46.704264 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.704252 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-istio-token" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:46.803587 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.803562 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-credential-socket\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.803735 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.803611 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ae9d702-f807-4040-b5e8-e140bead935f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae9d702-f807-4040-b5e8-e140bead935f-istio-podinfo\") pod \"8ae9d702-f807-4040-b5e8-e140bead935f\" (UID: \"8ae9d702-f807-4040-b5e8-e140bead935f\") " Oct 30 22:54:46.803831 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.803806 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-credential-socket" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:46.803973 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.803957 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.804025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.803974 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llx47\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-kube-api-access-llx47\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.804025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.803985 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae9d702-f807-4040-b5e8-e140bead935f-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.804025 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.803994 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae9d702-f807-4040-b5e8-e140bead935f-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:46.805585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.805564 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ae9d702-f807-4040-b5e8-e140bead935f-istio-podinfo" pod "8ae9d702-f807-4040-b5e8-e140bead935f" (UID: "8ae9d702-f807-4040-b5e8-e140bead935f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:54:46.905749 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:46.905717 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae9d702-f807-4040-b5e8-e140bead935f-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:47.071954 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.071922 4991 generic.go:356] "Generic (PLEG): container finished" podID="8ae9d702-f807-4040-b5e8-e140bead935f" containerID="26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e" exitCode=0 Oct 30 22:54:47.072131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.071973 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" Oct 30 22:54:47.072131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.072002 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" event={"ID":"8ae9d702-f807-4040-b5e8-e140bead935f","Type":"ContainerDied","Data":"26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e"} Oct 30 22:54:47.072131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.072062 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l" event={"ID":"8ae9d702-f807-4040-b5e8-e140bead935f","Type":"ContainerDied","Data":"18c92902a021d04ba293e57672f96535b5161abea0c957b6b082bdd2c7431063"} Oct 30 22:54:47.072131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.072094 4991 scope.go:122] "RemoveContainer" containerID="26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e" Oct 30 22:54:47.085266 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.085242 4991 scope.go:122] "RemoveContainer" containerID="0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283" Oct 30 22:54:47.094311 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.094293 4991 scope.go:122] "RemoveContainer" containerID="26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e" Oct 30 22:54:47.094604 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:54:47.094580 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e\": container with ID starting with 26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e not found: ID does not exist" containerID="26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e" Oct 30 22:54:47.094655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.094617 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e"} err="failed to get container status \"26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e\": rpc error: code = NotFound desc = could not find container \"26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e\": container with ID starting with 26e94ad4f99fd92d5a14c303fb614c278d7453ccde3870d006d3afc46cf5234e not found: ID does not exist" Oct 30 22:54:47.094655 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.094635 4991 scope.go:122] "RemoveContainer" containerID="0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283" Oct 30 22:54:47.094867 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:54:47.094848 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283\": container with ID starting with 0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283 not found: ID does not exist" containerID="0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283" Oct 30 22:54:47.094924 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.094878 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283"} err="failed to get container status \"0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283\": rpc error: code = NotFound desc = could not find container \"0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283\": container with ID starting with 0a4cafe085a0228ec4cc66bde8ea1ad19db4f339f4086cd58f9c9c348b315283 not found: ID does not exist" Oct 30 22:54:47.233526 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.233494 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l"] Oct 30 22:54:47.242950 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.242924 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5075-openshift-default-84f49cd6bf-ppv4l"] Oct 30 22:54:47.874967 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:54:47.874937 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" path="/var/lib/kubelet/pods/8ae9d702-f807-4040-b5e8-e140bead935f/volumes" Oct 30 22:54:47.910625 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:54:47.910588 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:55:05.543608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:05.543572 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4"] Oct 30 22:55:05.544064 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:05.543769 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" containerID="cri-o://7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8" gracePeriod=30 Oct 30 22:55:10.385835 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.385793 4991 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:55:10.386219 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.385859 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:55:10.792716 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.792693 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:55:10.941940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.941877 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istiod-ca-cert\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.941940 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-socket\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.941976 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-certs\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942011 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-data\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942083 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942050 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-envoy\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942082 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-kube-api-access-wnc9w\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-kube-api-access-wnc9w\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942113 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-token\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-token\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942225 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-podinfo\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942307 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942273 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-credential-socket\") pod \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\" (UID: \"8b4cc59e-42c4-4e5b-b996-2b4cf886879b\") " Oct 30 22:55:10.942563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942286 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-certs" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:10.942563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942398 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istiod-ca-cert" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:10.942563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942400 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-data" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:10.942563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942339 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-socket" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:10.942787 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942582 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-credential-socket" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:10.942855 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942839 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:10.942913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942864 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:10.942913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942881 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:10.942913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942895 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:10.942913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.942909 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:10.944449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.944423 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-envoy" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:10.944592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.944566 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-podinfo" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:55:10.944592 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.944568 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-token" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:10.944747 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:10.944667 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-kube-api-access-wnc9w" pod "8b4cc59e-42c4-4e5b-b996-2b4cf886879b" (UID: "8b4cc59e-42c4-4e5b-b996-2b4cf886879b"). InnerVolumeSpecName "kube-api-access-wnc9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:11.043795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.043762 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:11.043795 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.043791 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:11.043984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.043808 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc9w\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-kube-api-access-wnc9w\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:11.043984 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.043833 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4cc59e-42c4-4e5b-b996-2b4cf886879b-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:11.226140 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.226106 4991 generic.go:356] "Generic (PLEG): container finished" podID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerID="7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8" exitCode=0 Oct 30 22:55:11.226277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.226160 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" Oct 30 22:55:11.226277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.226186 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" event={"ID":"8b4cc59e-42c4-4e5b-b996-2b4cf886879b","Type":"ContainerDied","Data":"7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8"} Oct 30 22:55:11.226277 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.226251 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4" event={"ID":"8b4cc59e-42c4-4e5b-b996-2b4cf886879b","Type":"ContainerDied","Data":"c684a8c116a820ba43aaa657c687499129dd089f01952c7852162988b739de04"} Oct 30 22:55:11.226459 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.226277 4991 scope.go:122] "RemoveContainer" containerID="7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8" Oct 30 22:55:11.237154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.237132 4991 scope.go:122] "RemoveContainer" containerID="baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419" Oct 30 22:55:11.245833 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.245817 4991 scope.go:122] "RemoveContainer" containerID="7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8" Oct 30 22:55:11.246091 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:11.246072 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8\": container with ID starting with 7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8 not found: ID does not exist" containerID="7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8" Oct 30 22:55:11.246148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.246100 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8"} err="failed to get container status \"7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8\": rpc error: code = NotFound desc = could not find container \"7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8\": container with ID starting with 7ff7276ac8ee0ab934aa93452521f86390caf13decff03fe0b32f7c8e5bbbea8 not found: ID does not exist" Oct 30 22:55:11.246148 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.246126 4991 scope.go:122] "RemoveContainer" containerID="baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419" Oct 30 22:55:11.246388 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:11.246370 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419\": container with ID starting with baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419 not found: ID does not exist" containerID="baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419" Oct 30 22:55:11.246452 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.246395 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419"} err="failed to get container status \"baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419\": rpc error: code = NotFound desc = could not find container \"baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419\": container with ID starting with baa88d99d6f98f11965ed6e94ae10d3a759888692db07abd8027bfc01d04c419 not found: ID does not exist" Oct 30 22:55:11.384059 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.384026 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4"] Oct 30 22:55:11.430618 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.430589 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9717a8-openshift-default-6cb976d54f-wxmf4"] Oct 30 22:55:11.874693 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:11.874662 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" path="/var/lib/kubelet/pods/8b4cc59e-42c4-4e5b-b996-2b4cf886879b/volumes" Oct 30 22:55:13.078233 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.078203 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw"] Oct 30 22:55:13.078715 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.078477 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" podUID="9194f4ba-5aea-450b-8ac3-efefa4b1e743" containerName="maas-api" containerID="cri-o://8584b01fa4de5921d4aa1505c8b0b3017a72ee11407b9f7896b9f65417ea689f" gracePeriod=30 Oct 30 22:55:13.248741 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.248705 4991 generic.go:356] "Generic (PLEG): container finished" podID="9194f4ba-5aea-450b-8ac3-efefa4b1e743" containerID="8584b01fa4de5921d4aa1505c8b0b3017a72ee11407b9f7896b9f65417ea689f" exitCode=0 Oct 30 22:55:13.248945 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.248774 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" event={"ID":"9194f4ba-5aea-450b-8ac3-efefa4b1e743","Type":"ContainerDied","Data":"8584b01fa4de5921d4aa1505c8b0b3017a72ee11407b9f7896b9f65417ea689f"} Oct 30 22:55:13.361100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.361070 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 22:55:13.472712 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.472669 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-metrics-certs\") pod \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " Oct 30 22:55:13.472910 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.472805 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls\") pod \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " Oct 30 22:55:13.472910 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.472887 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9194f4ba-5aea-450b-8ac3-efefa4b1e743-kube-api-access-6c7nl\" (UniqueName: \"kubernetes.io/projected/9194f4ba-5aea-450b-8ac3-efefa4b1e743-kube-api-access-6c7nl\") pod \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " Oct 30 22:55:13.473073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.472950 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9194f4ba-5aea-450b-8ac3-efefa4b1e743-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9194f4ba-5aea-450b-8ac3-efefa4b1e743-openshift-service-ca\") pod \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\" (UID: \"9194f4ba-5aea-450b-8ac3-efefa4b1e743\") " Oct 30 22:55:13.473616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.473591 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9194f4ba-5aea-450b-8ac3-efefa4b1e743-openshift-service-ca" pod "9194f4ba-5aea-450b-8ac3-efefa4b1e743" (UID: "9194f4ba-5aea-450b-8ac3-efefa4b1e743"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:13.475510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.475453 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls" pod "9194f4ba-5aea-450b-8ac3-efefa4b1e743" (UID: "9194f4ba-5aea-450b-8ac3-efefa4b1e743"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:13.475624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.475555 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-metrics-certs" pod "9194f4ba-5aea-450b-8ac3-efefa4b1e743" (UID: "9194f4ba-5aea-450b-8ac3-efefa4b1e743"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:13.476753 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.476726 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9194f4ba-5aea-450b-8ac3-efefa4b1e743-kube-api-access-6c7nl" pod "9194f4ba-5aea-450b-8ac3-efefa4b1e743" (UID: "9194f4ba-5aea-450b-8ac3-efefa4b1e743"). InnerVolumeSpecName "kube-api-access-6c7nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:13.574668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.574627 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:13.574668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.574664 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9194f4ba-5aea-450b-8ac3-efefa4b1e743-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:13.574668 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.574674 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c7nl\" (UniqueName: \"kubernetes.io/projected/9194f4ba-5aea-450b-8ac3-efefa4b1e743-kube-api-access-6c7nl\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:13.574883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:13.574685 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9194f4ba-5aea-450b-8ac3-efefa4b1e743-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:14.255897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:14.255865 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" Oct 30 22:55:14.256487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:14.255896 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw" event={"ID":"9194f4ba-5aea-450b-8ac3-efefa4b1e743","Type":"ContainerDied","Data":"ad99e93399902542c599421ae658004b2d674155e19df96267341b5de5c4f421"} Oct 30 22:55:14.256487 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:14.255958 4991 scope.go:122] "RemoveContainer" containerID="8584b01fa4de5921d4aa1505c8b0b3017a72ee11407b9f7896b9f65417ea689f" Oct 30 22:55:14.412946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:14.412913 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw"] Oct 30 22:55:14.459739 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:14.459712 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b47e8d-8584bcb49d-qbrpw"] Oct 30 22:55:15.874961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:15.874927 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9194f4ba-5aea-450b-8ac3-efefa4b1e743" path="/var/lib/kubelet/pods/9194f4ba-5aea-450b-8ac3-efefa4b1e743/volumes" Oct 30 22:55:17.911790 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:17.911753 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:55:41.807842 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:41.807810 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27"] Oct 30 22:55:41.808254 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:41.808055 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" containerID="cri-o://03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376" gracePeriod=30 Oct 30 22:55:43.430105 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:43.430068 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs"] Oct 30 22:55:43.430590 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:43.430262 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" containerID="cri-o://53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a" gracePeriod=30 Oct 30 22:55:47.096748 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.096720 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:55:47.225808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.225771 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istiod-ca-cert\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.225808 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.225811 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-envoy\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.225842 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-podinfo\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.225870 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-certs\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.225910 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-credential-socket\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.225970 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-socket\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226043 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226010 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-data\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226348 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226065 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-token\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-token\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226348 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226145 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-kube-api-access-25nkg\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-kube-api-access-25nkg\") pod \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\" (UID: \"07a6c4db-a0b8-442c-90f4-52f3cfe31a29\") " Oct 30 22:55:47.226348 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226269 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-credential-socket" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:47.226348 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226306 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-socket" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:47.226591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226452 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-data" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:47.226591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226477 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istiod-ca-cert" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:47.226591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226526 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-certs" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:47.226757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226640 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.226757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226662 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.226757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226684 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.226757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226699 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.226757 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.226713 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.229051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.228751 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-token" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:47.229051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.228840 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-envoy" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:47.229051 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.228855 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-kube-api-access-25nkg" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "kube-api-access-25nkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:47.229337 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.229296 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-podinfo" pod "07a6c4db-a0b8-442c-90f4-52f3cfe31a29" (UID: "07a6c4db-a0b8-442c-90f4-52f3cfe31a29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:55:47.275301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.275274 4991 patch_prober.go:29] interesting pod/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:55:47.275458 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.275354 4991 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:55:47.327273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.327245 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25nkg\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-kube-api-access-25nkg\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.327273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.327273 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.327468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.327283 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.327468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.327291 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a6c4db-a0b8-442c-90f4-52f3cfe31a29-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:47.464410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.464336 4991 generic.go:356] "Generic (PLEG): container finished" podID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerID="03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376" exitCode=0 Oct 30 22:55:47.464410 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.464403 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" Oct 30 22:55:47.464626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.464425 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" event={"ID":"07a6c4db-a0b8-442c-90f4-52f3cfe31a29","Type":"ContainerDied","Data":"03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376"} Oct 30 22:55:47.464626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.464490 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27" event={"ID":"07a6c4db-a0b8-442c-90f4-52f3cfe31a29","Type":"ContainerDied","Data":"b0e309079e76dd691a2b773b681de00fb0c5c3bf15468a48b0fe09324a98fc87"} Oct 30 22:55:47.464626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.464522 4991 scope.go:122] "RemoveContainer" containerID="03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376" Oct 30 22:55:47.475002 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.474985 4991 scope.go:122] "RemoveContainer" containerID="06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab" Oct 30 22:55:47.484087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.484072 4991 scope.go:122] "RemoveContainer" containerID="03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376" Oct 30 22:55:47.484340 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:47.484311 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376\": container with ID starting with 03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376 not found: ID does not exist" containerID="03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376" Oct 30 22:55:47.484401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.484351 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376"} err="failed to get container status \"03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376\": rpc error: code = NotFound desc = could not find container \"03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376\": container with ID starting with 03de3920aa059beb785a5e03101a968b1a98020f372ca65c097d4c4bcab10376 not found: ID does not exist" Oct 30 22:55:47.484401 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.484366 4991 scope.go:122] "RemoveContainer" containerID="06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab" Oct 30 22:55:47.484580 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:47.484561 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab\": container with ID starting with 06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab not found: ID does not exist" containerID="06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab" Oct 30 22:55:47.484622 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.484587 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab"} err="failed to get container status \"06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab\": rpc error: code = NotFound desc = could not find container \"06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab\": container with ID starting with 06875a88fd8328090e1d5d2ef33d5c3e7e2b2ea3119e6584cc757899dccb60ab not found: ID does not exist" Oct 30 22:55:47.632830 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.632791 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27"] Oct 30 22:55:47.638299 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.638268 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9626a6-openshift-default-75b445487c-7lq27"] Oct 30 22:55:47.876488 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:47.876450 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" path="/var/lib/kubelet/pods/07a6c4db-a0b8-442c-90f4-52f3cfe31a29/volumes" Oct 30 22:55:47.916508 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:47.916471 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:55:48.687494 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.687470 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:55:48.841727 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841694 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-envoy\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.841923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841746 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-podinfo\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.841923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841788 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-credential-socket\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.841923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841815 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-token\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-token\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.841923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841835 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-data\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.841923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841859 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-socket\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.841923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841879 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-kube-api-access-r48nr\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-kube-api-access-r48nr\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.841923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841903 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-certs\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.842345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.841958 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istiod-ca-cert\") pod \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\" (UID: \"39de1aba-6701-42bf-bcfd-d2327f97f5a5\") " Oct 30 22:55:48.842345 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.842258 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-data" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:48.842484 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.842363 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.842913 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.842725 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-certs" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:48.843415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.843091 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-socket" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:48.843415 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.843149 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istiod-ca-cert" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:48.843673 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.843641 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-credential-socket" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:48.844380 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.844344 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-envoy" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:48.844791 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.844765 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-podinfo" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:55:48.845817 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.845787 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-token" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:48.846530 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.846512 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-kube-api-access-r48nr" pod "39de1aba-6701-42bf-bcfd-d2327f97f5a5" (UID: "39de1aba-6701-42bf-bcfd-d2327f97f5a5"). InnerVolumeSpecName "kube-api-access-r48nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:48.911585 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.911565 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm"] Oct 30 22:55:48.943605 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943578 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.943744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943608 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.943744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943623 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.943744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943636 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.943744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943653 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.943744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943668 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r48nr\" (UniqueName: \"kubernetes.io/projected/39de1aba-6701-42bf-bcfd-d2327f97f5a5-kube-api-access-r48nr\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.943744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943696 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39de1aba-6701-42bf-bcfd-d2327f97f5a5-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:48.943744 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:48.943711 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39de1aba-6701-42bf-bcfd-d2327f97f5a5-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:49.481281 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.481242 4991 generic.go:356] "Generic (PLEG): container finished" podID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerID="53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a" exitCode=0 Oct 30 22:55:49.481468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.481372 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" event={"ID":"39de1aba-6701-42bf-bcfd-d2327f97f5a5","Type":"ContainerDied","Data":"53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a"} Oct 30 22:55:49.481468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.481433 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" event={"ID":"39de1aba-6701-42bf-bcfd-d2327f97f5a5","Type":"ContainerDied","Data":"f224eb4040f68f646368e400507b4e59680311d784ad3b274d5b7b8519f55259"} Oct 30 22:55:49.481468 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.481457 4991 scope.go:122] "RemoveContainer" containerID="53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a" Oct 30 22:55:49.481613 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.481389 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs" Oct 30 22:55:49.492633 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.492460 4991 scope.go:122] "RemoveContainer" containerID="7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83" Oct 30 22:55:49.500882 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.500861 4991 scope.go:122] "RemoveContainer" containerID="53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a" Oct 30 22:55:49.501128 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:49.501106 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a\": container with ID starting with 53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a not found: ID does not exist" containerID="53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a" Oct 30 22:55:49.501188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.501135 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a"} err="failed to get container status \"53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a\": rpc error: code = NotFound desc = could not find container \"53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a\": container with ID starting with 53f03461c1299622f167d3c36ce749416517b4d04893d0b75b6c577c910ef87a not found: ID does not exist" Oct 30 22:55:49.501188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.501158 4991 scope.go:122] "RemoveContainer" containerID="7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83" Oct 30 22:55:49.501385 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:49.501365 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83\": container with ID starting with 7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83 not found: ID does not exist" containerID="7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83" Oct 30 22:55:49.501438 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.501392 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83"} err="failed to get container status \"7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83\": rpc error: code = NotFound desc = could not find container \"7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83\": container with ID starting with 7d431012aac4f8f7bc6bebc340571a8e9e227661012943fc203fbe5e86e6ff83 not found: ID does not exist" Oct 30 22:55:49.650985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.650947 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs"] Oct 30 22:55:49.701492 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.701465 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-68f80c-openshift-default-576bb9b6b7-bsjbs"] Oct 30 22:55:49.882958 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:49.882884 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" path="/var/lib/kubelet/pods/39de1aba-6701-42bf-bcfd-d2327f97f5a5/volumes" Oct 30 22:55:54.235194 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:54.235149 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf"] Oct 30 22:55:59.235985 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.235926 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" podUID="8a5f059b-0888-4996-9ab2-45bf7606cd44" containerName="payload-pre-processing" containerID="cri-o://cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544" gracePeriod=25 Oct 30 22:55:59.505391 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.505330 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:55:59.545185 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.545156 4991 generic.go:356] "Generic (PLEG): container finished" podID="8a5f059b-0888-4996-9ab2-45bf7606cd44" containerID="cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544" exitCode=0 Oct 30 22:55:59.545349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.545198 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" Oct 30 22:55:59.545349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.545240 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" event={"ID":"8a5f059b-0888-4996-9ab2-45bf7606cd44","Type":"ContainerDied","Data":"cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544"} Oct 30 22:55:59.545349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.545301 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf" event={"ID":"8a5f059b-0888-4996-9ab2-45bf7606cd44","Type":"ContainerDied","Data":"8210a00ad7c8a0502b2baf6e18e7120ca78d5eb8e725ba7fa6a6ba45033ceed8"} Oct 30 22:55:59.545349 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.545340 4991 scope.go:122] "RemoveContainer" containerID="cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544" Oct 30 22:55:59.558087 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.558067 4991 scope.go:122] "RemoveContainer" containerID="cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544" Oct 30 22:55:59.558362 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:55:59.558337 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544\": container with ID starting with cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544 not found: ID does not exist" containerID="cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544" Oct 30 22:55:59.558472 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.558369 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544"} err="failed to get container status \"cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544\": rpc error: code = NotFound desc = could not find container \"cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544\": container with ID starting with cfacd6f4f7a25e8ab8ee7f3be5f828928b267de65771bd212e9391dfc9ff0544 not found: ID does not exist" Oct 30 22:55:59.648905 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.648874 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5f059b-0888-4996-9ab2-45bf7606cd44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5f059b-0888-4996-9ab2-45bf7606cd44-plugins-config-volume\") pod \"8a5f059b-0888-4996-9ab2-45bf7606cd44\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " Oct 30 22:55:59.649072 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.648951 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5f059b-0888-4996-9ab2-45bf7606cd44-kube-api-access-l8z2f\" (UniqueName: \"kubernetes.io/projected/8a5f059b-0888-4996-9ab2-45bf7606cd44-kube-api-access-l8z2f\") pod \"8a5f059b-0888-4996-9ab2-45bf7606cd44\" (UID: \"8a5f059b-0888-4996-9ab2-45bf7606cd44\") " Oct 30 22:55:59.649278 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.649255 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5f059b-0888-4996-9ab2-45bf7606cd44-plugins-config-volume" pod "8a5f059b-0888-4996-9ab2-45bf7606cd44" (UID: "8a5f059b-0888-4996-9ab2-45bf7606cd44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:59.649390 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.649371 4991 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5f059b-0888-4996-9ab2-45bf7606cd44-plugins-config-volume\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:59.651085 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.651058 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5f059b-0888-4996-9ab2-45bf7606cd44-kube-api-access-l8z2f" pod "8a5f059b-0888-4996-9ab2-45bf7606cd44" (UID: "8a5f059b-0888-4996-9ab2-45bf7606cd44"). InnerVolumeSpecName "kube-api-access-l8z2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:59.750236 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.750207 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8z2f\" (UniqueName: \"kubernetes.io/projected/8a5f059b-0888-4996-9ab2-45bf7606cd44-kube-api-access-l8z2f\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:59.976249 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.976222 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf"] Oct 30 22:55:59.981256 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:55:59.981231 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3b871a-6b96b7c47f-j5pjf"] Oct 30 22:56:01.880490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:01.880451 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5f059b-0888-4996-9ab2-45bf7606cd44" path="/var/lib/kubelet/pods/8a5f059b-0888-4996-9ab2-45bf7606cd44/volumes" Oct 30 22:56:03.508055 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.508019 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6"] Oct 30 22:56:03.570412 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.570373 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" podUID="66834ead-f417-4228-bd5f-a5cde6ca346f" containerName="maas-api" containerID="cri-o://b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c" gracePeriod=30 Oct 30 22:56:03.832161 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.832123 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:56:03.932188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.932139 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" containerID="cri-o://60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de" gracePeriod=45 Oct 30 22:56:03.989932 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.989890 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-metrics-certs\") pod \"66834ead-f417-4228-bd5f-a5cde6ca346f\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " Oct 30 22:56:03.990086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.990021 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66834ead-f417-4228-bd5f-a5cde6ca346f-kube-api-access-7g627\" (UniqueName: \"kubernetes.io/projected/66834ead-f417-4228-bd5f-a5cde6ca346f-kube-api-access-7g627\") pod \"66834ead-f417-4228-bd5f-a5cde6ca346f\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " Oct 30 22:56:03.990086 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.990048 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls\") pod \"66834ead-f417-4228-bd5f-a5cde6ca346f\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " Oct 30 22:56:03.990212 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.990184 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66834ead-f417-4228-bd5f-a5cde6ca346f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66834ead-f417-4228-bd5f-a5cde6ca346f-openshift-service-ca\") pod \"66834ead-f417-4228-bd5f-a5cde6ca346f\" (UID: \"66834ead-f417-4228-bd5f-a5cde6ca346f\") " Oct 30 22:56:03.990598 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.990574 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66834ead-f417-4228-bd5f-a5cde6ca346f-openshift-service-ca" pod "66834ead-f417-4228-bd5f-a5cde6ca346f" (UID: "66834ead-f417-4228-bd5f-a5cde6ca346f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:03.990732 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.990687 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66834ead-f417-4228-bd5f-a5cde6ca346f-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:03.992125 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.992102 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-metrics-certs" pod "66834ead-f417-4228-bd5f-a5cde6ca346f" (UID: "66834ead-f417-4228-bd5f-a5cde6ca346f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:03.992213 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.992133 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66834ead-f417-4228-bd5f-a5cde6ca346f-kube-api-access-7g627" pod "66834ead-f417-4228-bd5f-a5cde6ca346f" (UID: "66834ead-f417-4228-bd5f-a5cde6ca346f"). InnerVolumeSpecName "kube-api-access-7g627". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:03.992213 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:03.992138 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls" pod "66834ead-f417-4228-bd5f-a5cde6ca346f" (UID: "66834ead-f417-4228-bd5f-a5cde6ca346f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:04.092283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.092150 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g627\" (UniqueName: \"kubernetes.io/projected/66834ead-f417-4228-bd5f-a5cde6ca346f-kube-api-access-7g627\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.092283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.092184 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.092283 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.092198 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66834ead-f417-4228-bd5f-a5cde6ca346f-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.181354 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.181307 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:56:04.294428 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294397 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6309d2d6-8637-4a0f-9520-7024b8957e5e-kube-api-access-7g47v\" (UniqueName: \"kubernetes.io/projected/6309d2d6-8637-4a0f-9520-7024b8957e5e-kube-api-access-7g47v\") pod \"6309d2d6-8637-4a0f-9520-7024b8957e5e\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " Oct 30 22:56:04.294606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294493 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-tmp-dir\") pod \"6309d2d6-8637-4a0f-9520-7024b8957e5e\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " Oct 30 22:56:04.294606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294524 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-dshm\") pod \"6309d2d6-8637-4a0f-9520-7024b8957e5e\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " Oct 30 22:56:04.294606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294558 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-model-cache\") pod \"6309d2d6-8637-4a0f-9520-7024b8957e5e\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " Oct 30 22:56:04.294606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294606 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6309d2d6-8637-4a0f-9520-7024b8957e5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6309d2d6-8637-4a0f-9520-7024b8957e5e-tls-certs\") pod \"6309d2d6-8637-4a0f-9520-7024b8957e5e\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " Oct 30 22:56:04.294858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294653 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-home\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-home\") pod \"6309d2d6-8637-4a0f-9520-7024b8957e5e\" (UID: \"6309d2d6-8637-4a0f-9520-7024b8957e5e\") " Oct 30 22:56:04.294858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294777 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-tmp-dir" pod "6309d2d6-8637-4a0f-9520-7024b8957e5e" (UID: "6309d2d6-8637-4a0f-9520-7024b8957e5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:04.294962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294876 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-model-cache" pod "6309d2d6-8637-4a0f-9520-7024b8957e5e" (UID: "6309d2d6-8637-4a0f-9520-7024b8957e5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:04.294962 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294929 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-home" pod "6309d2d6-8637-4a0f-9520-7024b8957e5e" (UID: "6309d2d6-8637-4a0f-9520-7024b8957e5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:04.295058 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.294999 4991 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-tmp-dir\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.295058 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.295016 4991 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-model-cache\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.295058 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.295026 4991 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-home\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.297332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.297289 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-dshm" pod "6309d2d6-8637-4a0f-9520-7024b8957e5e" (UID: "6309d2d6-8637-4a0f-9520-7024b8957e5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:04.297332 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.297306 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6309d2d6-8637-4a0f-9520-7024b8957e5e-tls-certs" pod "6309d2d6-8637-4a0f-9520-7024b8957e5e" (UID: "6309d2d6-8637-4a0f-9520-7024b8957e5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:04.297523 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.297494 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6309d2d6-8637-4a0f-9520-7024b8957e5e-kube-api-access-7g47v" pod "6309d2d6-8637-4a0f-9520-7024b8957e5e" (UID: "6309d2d6-8637-4a0f-9520-7024b8957e5e"). InnerVolumeSpecName "kube-api-access-7g47v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:04.396059 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.395978 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g47v\" (UniqueName: \"kubernetes.io/projected/6309d2d6-8637-4a0f-9520-7024b8957e5e-kube-api-access-7g47v\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.396059 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.396011 4991 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6309d2d6-8637-4a0f-9520-7024b8957e5e-dshm\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.396059 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.396020 4991 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6309d2d6-8637-4a0f-9520-7024b8957e5e-tls-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:04.577396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.577357 4991 generic.go:356] "Generic (PLEG): container finished" podID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerID="60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de" exitCode=0 Oct 30 22:56:04.577396 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.577399 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" Oct 30 22:56:04.577875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.577433 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" event={"ID":"6309d2d6-8637-4a0f-9520-7024b8957e5e","Type":"ContainerDied","Data":"60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de"} Oct 30 22:56:04.577875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.577479 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm" event={"ID":"6309d2d6-8637-4a0f-9520-7024b8957e5e","Type":"ContainerDied","Data":"88c40827df15997f56617972514e1036d1c494280b16053d5da29872f00fccb8"} Oct 30 22:56:04.577875 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.577495 4991 scope.go:122] "RemoveContainer" containerID="60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de" Oct 30 22:56:04.579126 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.579104 4991 generic.go:356] "Generic (PLEG): container finished" podID="66834ead-f417-4228-bd5f-a5cde6ca346f" containerID="b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c" exitCode=0 Oct 30 22:56:04.579230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.579149 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" Oct 30 22:56:04.579230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.579173 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" event={"ID":"66834ead-f417-4228-bd5f-a5cde6ca346f","Type":"ContainerDied","Data":"b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c"} Oct 30 22:56:04.579230 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.579216 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6" event={"ID":"66834ead-f417-4228-bd5f-a5cde6ca346f","Type":"ContainerDied","Data":"33c1ea3b9f5fff2c560813d4d4031a802a30506e58492550f2e7599346385608"} Oct 30 22:56:04.604114 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.604088 4991 scope.go:122] "RemoveContainer" containerID="60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de" Oct 30 22:56:04.604484 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:04.604461 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de\": container with ID starting with 60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de not found: ID does not exist" containerID="60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de" Oct 30 22:56:04.604571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.604494 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de"} err="failed to get container status \"60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de\": rpc error: code = NotFound desc = could not find container \"60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de\": container with ID starting with 60aec550f97b462c1ea914949d1aad6b2c8cd4aac98a7151fe7413f9e0f744de not found: ID does not exist" Oct 30 22:56:04.604571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.604514 4991 scope.go:122] "RemoveContainer" containerID="b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c" Oct 30 22:56:04.613608 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.613583 4991 scope.go:122] "RemoveContainer" containerID="b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c" Oct 30 22:56:04.613916 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:04.613892 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c\": container with ID starting with b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c not found: ID does not exist" containerID="b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c" Oct 30 22:56:04.614003 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.613922 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c"} err="failed to get container status \"b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c\": rpc error: code = NotFound desc = could not find container \"b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c\": container with ID starting with b451520c6cd1fba4d32cc1ef5f420e3d4116314546d4ba69c1b80218cbd7ef6c not found: ID does not exist" Oct 30 22:56:04.746531 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.746495 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6"] Oct 30 22:56:04.750247 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.750219 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3b871a-6cfc66dc-fnsw6"] Oct 30 22:56:04.859056 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.859021 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm"] Oct 30 22:56:04.865887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:04.865857 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-w2-7f66ab-kserve-79d6c48dcc-bd8sm"] Oct 30 22:56:05.873979 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:05.873946 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" path="/var/lib/kubelet/pods/6309d2d6-8637-4a0f-9520-7024b8957e5e/volumes" Oct 30 22:56:05.874393 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:05.874377 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66834ead-f417-4228-bd5f-a5cde6ca346f" path="/var/lib/kubelet/pods/66834ead-f417-4228-bd5f-a5cde6ca346f/volumes" Oct 30 22:56:13.867961 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:13.867922 4991 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" secret="" err="secret \"e2e-extmod-b-3b871a-openshift-default-dockercfg-s9764\" not found" Oct 30 22:56:13.908653 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:13.908621 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9"] Oct 30 22:56:14.640132 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:14.640072 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" containerID="cri-o://14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4" gracePeriod=30 Oct 30 22:56:17.911449 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:17.911414 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:56:19.907907 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:19.907852 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:56:20.042616 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.042582 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-envoy\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.042773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.042636 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-podinfo\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.042773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.042680 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-socket\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.042773 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.042766 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77b4f03-6347-4591-919d-0aa3ba2453fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77b4f03-6347-4591-919d-0aa3ba2453fc-istiod-ca-cert\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.042951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.042811 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-certs\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.042951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.042868 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-credential-socket\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.042951 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.042904 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-token\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-token\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.043134 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043042 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-socket" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:20.043259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043230 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-data" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:20.043419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043268 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-data\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.043419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043362 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-kube-api-access-66hgj\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-kube-api-access-66hgj\") pod \"e77b4f03-6347-4591-919d-0aa3ba2453fc\" (UID: \"e77b4f03-6347-4591-919d-0aa3ba2453fc\") " Oct 30 22:56:20.043419 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043411 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-certs" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:20.043705 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043685 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.043785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043708 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.043785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043489 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77b4f03-6347-4591-919d-0aa3ba2453fc-istiod-ca-cert" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:20.043785 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.043761 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-credential-socket" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:20.044983 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.044954 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-envoy" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:20.045270 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.045242 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-podinfo" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:56:20.045570 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.045547 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-kube-api-access-66hgj" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "kube-api-access-66hgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:20.046290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.046263 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-token" pod "e77b4f03-6347-4591-919d-0aa3ba2453fc" (UID: "e77b4f03-6347-4591-919d-0aa3ba2453fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:20.144955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.144925 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.144955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.144950 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.144955 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.144961 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77b4f03-6347-4591-919d-0aa3ba2453fc-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.145154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.144969 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.145154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.144978 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77b4f03-6347-4591-919d-0aa3ba2453fc-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.145154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.144987 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.145154 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.144996 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66hgj\" (UniqueName: \"kubernetes.io/projected/e77b4f03-6347-4591-919d-0aa3ba2453fc-kube-api-access-66hgj\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.617359 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.617294 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:56:20.630193 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.630159 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jqbm4_04efc6e4-19af-4935-a7c1-2dac67498e4c/console-operator/1.log" Oct 30 22:56:20.681262 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.681231 4991 generic.go:356] "Generic (PLEG): container finished" podID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerID="14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4" exitCode=0 Oct 30 22:56:20.681420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.681283 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" Oct 30 22:56:20.681420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.681344 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" event={"ID":"e77b4f03-6347-4591-919d-0aa3ba2453fc","Type":"ContainerDied","Data":"14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4"} Oct 30 22:56:20.681420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.681396 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9" event={"ID":"e77b4f03-6347-4591-919d-0aa3ba2453fc","Type":"ContainerDied","Data":"be1a6088258b0159f9fe8604c62e28a6e11aecea9c6fd91232abd463203eaf4f"} Oct 30 22:56:20.681420 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.681418 4991 scope.go:122] "RemoveContainer" containerID="14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4" Oct 30 22:56:20.691571 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.691553 4991 scope.go:122] "RemoveContainer" containerID="1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7" Oct 30 22:56:20.699922 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.699906 4991 scope.go:122] "RemoveContainer" containerID="14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4" Oct 30 22:56:20.700168 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:20.700152 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4\": container with ID starting with 14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4 not found: ID does not exist" containerID="14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4" Oct 30 22:56:20.700228 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.700174 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4"} err="failed to get container status \"14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4\": rpc error: code = NotFound desc = could not find container \"14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4\": container with ID starting with 14002523488fe71bbb9eb1eff04948fb3121cf25df2ea6a31574ce929218d7e4 not found: ID does not exist" Oct 30 22:56:20.700228 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.700188 4991 scope.go:122] "RemoveContainer" containerID="1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7" Oct 30 22:56:20.700435 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:20.700419 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7\": container with ID starting with 1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7 not found: ID does not exist" containerID="1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7" Oct 30 22:56:20.700510 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.700439 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7"} err="failed to get container status \"1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7\": rpc error: code = NotFound desc = could not find container \"1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7\": container with ID starting with 1d7e48e313328b454155e30b3e073cd2ac7871a68c8cd18af714a96a8a716fa7 not found: ID does not exist" Oct 30 22:56:20.848373 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.848343 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9"] Oct 30 22:56:20.852812 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:20.852791 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3b871a-openshift-default-84d65fc85c-lrxk9"] Oct 30 22:56:21.873386 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:21.873275 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" path="/var/lib/kubelet/pods/e77b4f03-6347-4591-919d-0aa3ba2453fc/volumes" Oct 30 22:56:23.644627 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.644595 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t"] Oct 30 22:56:23.645080 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.644780 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" podUID="a5d5ff2f-96f2-4477-b319-219840d3db21" containerName="maas-api" containerID="cri-o://57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24" gracePeriod=30 Oct 30 22:56:23.893745 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.893718 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 22:56:23.978607 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.978565 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d5ff2f-96f2-4477-b319-219840d3db21-kube-api-access-v2xm4\" (UniqueName: \"kubernetes.io/projected/a5d5ff2f-96f2-4477-b319-219840d3db21-kube-api-access-v2xm4\") pod \"a5d5ff2f-96f2-4477-b319-219840d3db21\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " Oct 30 22:56:23.978778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.978641 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d5ff2f-96f2-4477-b319-219840d3db21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5ff2f-96f2-4477-b319-219840d3db21-openshift-service-ca\") pod \"a5d5ff2f-96f2-4477-b319-219840d3db21\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " Oct 30 22:56:23.978778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.978688 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls\") pod \"a5d5ff2f-96f2-4477-b319-219840d3db21\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " Oct 30 22:56:23.978778 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.978733 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-metrics-certs\") pod \"a5d5ff2f-96f2-4477-b319-219840d3db21\" (UID: \"a5d5ff2f-96f2-4477-b319-219840d3db21\") " Oct 30 22:56:23.978980 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.978924 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d5ff2f-96f2-4477-b319-219840d3db21-openshift-service-ca" pod "a5d5ff2f-96f2-4477-b319-219840d3db21" (UID: "a5d5ff2f-96f2-4477-b319-219840d3db21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:23.979088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.979072 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5ff2f-96f2-4477-b319-219840d3db21-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:23.980802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.980775 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d5ff2f-96f2-4477-b319-219840d3db21-kube-api-access-v2xm4" pod "a5d5ff2f-96f2-4477-b319-219840d3db21" (UID: "a5d5ff2f-96f2-4477-b319-219840d3db21"). InnerVolumeSpecName "kube-api-access-v2xm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:23.980802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.980795 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls" pod "a5d5ff2f-96f2-4477-b319-219840d3db21" (UID: "a5d5ff2f-96f2-4477-b319-219840d3db21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:23.980974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:23.980876 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-metrics-certs" pod "a5d5ff2f-96f2-4477-b319-219840d3db21" (UID: "a5d5ff2f-96f2-4477-b319-219840d3db21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:24.080235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.080198 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2xm4\" (UniqueName: \"kubernetes.io/projected/a5d5ff2f-96f2-4477-b319-219840d3db21-kube-api-access-v2xm4\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:24.080235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.080227 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:24.080235 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.080238 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d5ff2f-96f2-4477-b319-219840d3db21-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:24.708678 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.708643 4991 generic.go:356] "Generic (PLEG): container finished" podID="a5d5ff2f-96f2-4477-b319-219840d3db21" containerID="57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24" exitCode=0 Oct 30 22:56:24.709092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.708689 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" Oct 30 22:56:24.709092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.708723 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" event={"ID":"a5d5ff2f-96f2-4477-b319-219840d3db21","Type":"ContainerDied","Data":"57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24"} Oct 30 22:56:24.709092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.708763 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t" event={"ID":"a5d5ff2f-96f2-4477-b319-219840d3db21","Type":"ContainerDied","Data":"c4e3f2da5994537d6dc967abcdc7d1e4b6fcbb598689b0c36d20ddf02a83ab4c"} Oct 30 22:56:24.709092 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.708784 4991 scope.go:122] "RemoveContainer" containerID="57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24" Oct 30 22:56:24.719698 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.719672 4991 scope.go:122] "RemoveContainer" containerID="57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24" Oct 30 22:56:24.719930 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:24.719910 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24\": container with ID starting with 57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24 not found: ID does not exist" containerID="57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24" Oct 30 22:56:24.719981 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.719939 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24"} err="failed to get container status \"57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24\": rpc error: code = NotFound desc = could not find container \"57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24\": container with ID starting with 57651bf0de9bdafc561ab4e17a527f8bdb5687110d6707004e44c5344eb9ec24 not found: ID does not exist" Oct 30 22:56:24.878520 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.878485 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t"] Oct 30 22:56:24.928776 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:24.928747 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7f66ab-6c94d6cf79-jqk7t"] Oct 30 22:56:25.874906 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:25.874875 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d5ff2f-96f2-4477-b319-219840d3db21" path="/var/lib/kubelet/pods/a5d5ff2f-96f2-4477-b319-219840d3db21/volumes" Oct 30 22:56:47.911959 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:47.911924 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:56:53.571119 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:53.571088 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j"] Oct 30 22:56:53.571624 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:53.571307 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" containerID="cri-o://750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4" gracePeriod=30 Oct 30 22:56:58.823563 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.823536 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:56:58.886290 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886235 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-envoy\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886300 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-token\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-token\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886346 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-data\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886432 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886383 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-certs\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886430 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-socket\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886474 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istiod-ca-cert\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886517 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-kube-api-access-6kzvp\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-kube-api-access-6kzvp\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886547 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-credential-socket\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886611 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886578 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-podinfo\") pod \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\" (UID: \"843f57c2-8883-4f2a-aec5-3e8e940ad5fd\") " Oct 30 22:56:58.886884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886685 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-data" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:58.886884 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886735 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-certs" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:58.887008 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.886880 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istiod-ca-cert" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:58.887066 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.887022 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-socket" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:58.887197 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.887177 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-credential-socket" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:58.887572 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.887504 4991 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istiod-ca-cert\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.887572 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.887528 4991 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-data\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.887572 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.887546 4991 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.888626 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.888606 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-token" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:58.888859 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.888837 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-envoy" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:58.888957 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.888941 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-podinfo" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:56:58.889186 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.889167 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-kube-api-access-6kzvp" pod "843f57c2-8883-4f2a-aec5-3e8e940ad5fd" (UID: "843f57c2-8883-4f2a-aec5-3e8e940ad5fd"). InnerVolumeSpecName "kube-api-access-6kzvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:58.904363 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.904339 4991 generic.go:356] "Generic (PLEG): container finished" podID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerID="750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4" exitCode=0 Oct 30 22:56:58.904449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.904383 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" Oct 30 22:56:58.904449 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.904417 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" event={"ID":"843f57c2-8883-4f2a-aec5-3e8e940ad5fd","Type":"ContainerDied","Data":"750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4"} Oct 30 22:56:58.904533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.904466 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j" event={"ID":"843f57c2-8883-4f2a-aec5-3e8e940ad5fd","Type":"ContainerDied","Data":"b1c1758abf621a6984c6796bb0366469b8db25a31cb3dae0c296171cffca9474"} Oct 30 22:56:58.904533 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.904487 4991 scope.go:122] "RemoveContainer" containerID="750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4" Oct 30 22:56:58.913992 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.913970 4991 scope.go:122] "RemoveContainer" containerID="ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba" Oct 30 22:56:58.922269 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.922251 4991 scope.go:122] "RemoveContainer" containerID="750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4" Oct 30 22:56:58.922564 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:58.922546 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4\": container with ID starting with 750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4 not found: ID does not exist" containerID="750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4" Oct 30 22:56:58.922649 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.922572 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4"} err="failed to get container status \"750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4\": rpc error: code = NotFound desc = could not find container \"750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4\": container with ID starting with 750e2ad027ac79513eb91120bd09b0846091234c24991bd0a3a764472ae945f4 not found: ID does not exist" Oct 30 22:56:58.922649 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.922590 4991 scope.go:122] "RemoveContainer" containerID="ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba" Oct 30 22:56:58.922832 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:56:58.922814 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba\": container with ID starting with ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba not found: ID does not exist" containerID="ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba" Oct 30 22:56:58.922885 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.922837 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba"} err="failed to get container status \"ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba\": rpc error: code = NotFound desc = could not find container \"ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba\": container with ID starting with ebeb6fc560a894f886361819555b23fdf51f8c918ae4ae6cc92bae334a9958ba not found: ID does not exist" Oct 30 22:56:58.988141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.988115 4991 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-workload-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.988141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.988138 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzvp\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-kube-api-access-6kzvp\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.988273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.988150 4991 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-credential-socket\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.988273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.988159 4991 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-podinfo\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.988273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.988167 4991 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-envoy\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:58.988273 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:58.988176 4991 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843f57c2-8883-4f2a-aec5-3e8e940ad5fd-istio-token\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:59.109731 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:59.109705 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j"] Oct 30 22:56:59.113492 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:59.113467 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b392d-openshift-default-7559dd76bf-zlj8j"] Oct 30 22:56:59.874695 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:56:59.874660 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" path="/var/lib/kubelet/pods/843f57c2-8883-4f2a-aec5-3e8e940ad5fd/volumes" Oct 30 22:57:17.911514 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:57:17.911476 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:57:47.910308 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:57:47.910270 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:57:53.619175 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.619139 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985"] Oct 30 22:57:53.619756 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.619336 4991 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" podUID="b639357e-4f63-439a-b1b6-2e32bfa296a2" containerName="maas-api" containerID="cri-o://226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6" gracePeriod=30 Oct 30 22:57:53.878118 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.878059 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 22:57:53.954532 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.954506 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-metrics-certs\") pod \"b639357e-4f63-439a-b1b6-2e32bfa296a2\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " Oct 30 22:57:53.954691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.954619 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b639357e-4f63-439a-b1b6-2e32bfa296a2-kube-api-access-zjm4j\" (UniqueName: \"kubernetes.io/projected/b639357e-4f63-439a-b1b6-2e32bfa296a2-kube-api-access-zjm4j\") pod \"b639357e-4f63-439a-b1b6-2e32bfa296a2\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " Oct 30 22:57:53.954691 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.954660 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b639357e-4f63-439a-b1b6-2e32bfa296a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b639357e-4f63-439a-b1b6-2e32bfa296a2-openshift-service-ca\") pod \"b639357e-4f63-439a-b1b6-2e32bfa296a2\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " Oct 30 22:57:53.954811 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.954714 4991 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls\") pod \"b639357e-4f63-439a-b1b6-2e32bfa296a2\" (UID: \"b639357e-4f63-439a-b1b6-2e32bfa296a2\") " Oct 30 22:57:53.955023 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.955000 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b639357e-4f63-439a-b1b6-2e32bfa296a2-openshift-service-ca" pod "b639357e-4f63-439a-b1b6-2e32bfa296a2" (UID: "b639357e-4f63-439a-b1b6-2e32bfa296a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:53.956631 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.956608 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-metrics-certs" pod "b639357e-4f63-439a-b1b6-2e32bfa296a2" (UID: "b639357e-4f63-439a-b1b6-2e32bfa296a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:53.956764 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.956654 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b639357e-4f63-439a-b1b6-2e32bfa296a2-kube-api-access-zjm4j" pod "b639357e-4f63-439a-b1b6-2e32bfa296a2" (UID: "b639357e-4f63-439a-b1b6-2e32bfa296a2"). InnerVolumeSpecName "kube-api-access-zjm4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:53.956813 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:53.956776 4991 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls" pod "b639357e-4f63-439a-b1b6-2e32bfa296a2" (UID: "b639357e-4f63-439a-b1b6-2e32bfa296a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:54.055786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.055755 4991 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm4j\" (UniqueName: \"kubernetes.io/projected/b639357e-4f63-439a-b1b6-2e32bfa296a2-kube-api-access-zjm4j\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:54.055786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.055780 4991 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b639357e-4f63-439a-b1b6-2e32bfa296a2-openshift-service-ca\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:54.055786 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.055791 4991 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-maas-api-tls\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:54.055987 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.055800 4991 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b639357e-4f63-439a-b1b6-2e32bfa296a2-metrics-certs\") on node \"ip-10-0-137-17.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:54.199658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.199582 4991 generic.go:356] "Generic (PLEG): container finished" podID="b639357e-4f63-439a-b1b6-2e32bfa296a2" containerID="226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6" exitCode=0 Oct 30 22:57:54.199658 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.199631 4991 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" Oct 30 22:57:54.199828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.199672 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" event={"ID":"b639357e-4f63-439a-b1b6-2e32bfa296a2","Type":"ContainerDied","Data":"226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6"} Oct 30 22:57:54.199828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.199730 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985" event={"ID":"b639357e-4f63-439a-b1b6-2e32bfa296a2","Type":"ContainerDied","Data":"9301942c6eba1c91e73975b613995c6198ed39507c91736bdc3e2a9878e84767"} Oct 30 22:57:54.199828 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.199759 4991 scope.go:122] "RemoveContainer" containerID="226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6" Oct 30 22:57:54.214082 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.214060 4991 scope.go:122] "RemoveContainer" containerID="226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6" Oct 30 22:57:54.214630 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:57:54.214603 4991 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6\": container with ID starting with 226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6 not found: ID does not exist" containerID="226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6" Oct 30 22:57:54.214752 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.214640 4991 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6"} err="failed to get container status \"226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6\": rpc error: code = NotFound desc = could not find container \"226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6\": container with ID starting with 226bac15507fdd4cf3a764df6ca73d0655a4f508fdb67d5e9ab69d78b3cf66b6 not found: ID does not exist" Oct 30 22:57:54.367136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.367109 4991 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985"] Oct 30 22:57:54.370000 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:54.369980 4991 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d09fff-5f8645df64-6w985"] Oct 30 22:57:55.878733 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:57:55.877530 4991 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b639357e-4f63-439a-b1b6-2e32bfa296a2" path="/var/lib/kubelet/pods/b639357e-4f63-439a-b1b6-2e32bfa296a2/volumes" Oct 30 22:58:04.145039 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:04.145000 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57dbf74b78-dx9mj_c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd/authorino/0.log" Oct 30 22:58:04.539918 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:04.539883 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdbtw_98b55f48-d625-4a7f-bbd7-98e462d435b5/manager/0.log" Oct 30 22:58:04.786883 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:04.786847 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c8sn_0ec987ff-0828-4a26-8926-874d8a47cbc7/manager/0.log" Oct 30 22:58:06.225291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:06.225256 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhmj8_0bf5a041-29ee-46b9-88ae-a1bfd29d24c2/manager/0.log" Oct 30 22:58:09.104490 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:09.104450 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:58:09.352045 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:09.352009 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:58:10.664738 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:10.664706 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57dbf74b78-dx9mj_c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd/authorino/0.log" Oct 30 22:58:17.909697 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:58:17.909660 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:58:22.705589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:22.705549 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:58:22.958218 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:22.958189 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:58:24.284937 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:24.284903 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-844dd57557-7jtjc_868fdf52-6fd4-4c94-871c-313badd2fb38/maas-api/0.log" Oct 30 22:58:27.916897 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:27.916866 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57dbf74b78-dx9mj_c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd/authorino/0.log" Oct 30 22:58:28.172377 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:28.172277 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdbtw_98b55f48-d625-4a7f-bbd7-98e462d435b5/manager/0.log" Oct 30 22:58:28.414131 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:28.414094 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c8sn_0ec987ff-0828-4a26-8926-874d8a47cbc7/manager/0.log" Oct 30 22:58:29.505469 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:29.505433 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhmj8_0bf5a041-29ee-46b9-88ae-a1bfd29d24c2/manager/0.log" Oct 30 22:58:30.936330 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:30.936285 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz_e0038133-9d86-4cf5-a75f-539470632d85/istio-proxy/0.log" Oct 30 22:58:32.089940 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:32.089910 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph_ec12978d-ab70-452f-8b7b-4954fa7fce4c/istio-proxy/0.log" Oct 30 22:58:32.772758 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:32.772726 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8c9db44f-llgx4_a547a8cd-8909-475e-a03d-15b409877777/router/0.log" Oct 30 22:58:33.913925 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:33.913895 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kcnm8_0e5b737c-58c1-4ef3-a674-62eb4eb1447e/main/0.log" Oct 30 22:58:34.652867 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:34.652835 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s2xqp_a617761d-3aee-4bf5-84ef-c98a111da859/main/0.log" Oct 30 22:58:35.188964 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:35.188935 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8xstj_1175e1de-ccd9-486e-904e-ca0d060232b7/main/0.log" Oct 30 22:58:35.469395 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:35.469311 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2kn7t_227fcbbf-d027-4db6-b8ec-03659183f431/main/0.log" Oct 30 22:58:45.491427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491393 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mpzc/must-gather-8nz7d"] Oct 30 22:58:45.491802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491771 4991 cpu_manager.go:414] "containerMap: removing container" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" Oct 30 22:58:45.491802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491783 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" Oct 30 22:58:45.491802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491795 4991 cpu_manager.go:414] "containerMap: removing container" podUID="9194f4ba-5aea-450b-8ac3-efefa4b1e743" containerName="maas-api" Oct 30 22:58:45.491802 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491800 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9194f4ba-5aea-450b-8ac3-efefa4b1e743" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491808 4991 cpu_manager.go:414] "containerMap: removing container" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491814 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491822 4991 cpu_manager.go:414] "containerMap: removing container" podUID="92d4fd17-1756-47a6-81df-e3d7174555f0" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491827 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d4fd17-1756-47a6-81df-e3d7174555f0" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491834 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491839 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491845 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491850 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491857 4991 cpu_manager.go:414] "containerMap: removing container" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491862 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491869 4991 cpu_manager.go:414] "containerMap: removing container" podUID="cbacae8f-ac18-44ba-aedb-6ecb049209dc" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491873 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbacae8f-ac18-44ba-aedb-6ecb049209dc" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491883 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491887 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491893 4991 cpu_manager.go:414] "containerMap: removing container" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491898 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491903 4991 cpu_manager.go:414] "containerMap: removing container" podUID="b639357e-4f63-439a-b1b6-2e32bfa296a2" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491907 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b639357e-4f63-439a-b1b6-2e32bfa296a2" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491916 4991 cpu_manager.go:414] "containerMap: removing container" podUID="92343dce-b0c6-44f8-a913-e718e6eb0ad7" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491921 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92343dce-b0c6-44f8-a913-e718e6eb0ad7" containerName="maas-api" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491931 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491935 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491944 4991 cpu_manager.go:414] "containerMap: removing container" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491949 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" Oct 30 22:58:45.491946 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491956 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491960 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491967 4991 cpu_manager.go:414] "containerMap: removing container" podUID="66834ead-f417-4228-bd5f-a5cde6ca346f" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491971 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66834ead-f417-4228-bd5f-a5cde6ca346f" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491978 4991 cpu_manager.go:414] "containerMap: removing container" podUID="a5d5ff2f-96f2-4477-b319-219840d3db21" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491983 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d5ff2f-96f2-4477-b319-219840d3db21" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491992 4991 cpu_manager.go:414] "containerMap: removing container" podUID="8a5f059b-0888-4996-9ab2-45bf7606cd44" containerName="payload-pre-processing" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.491996 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5f059b-0888-4996-9ab2-45bf7606cd44" containerName="payload-pre-processing" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492004 4991 cpu_manager.go:414] "containerMap: removing container" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492009 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492016 4991 cpu_manager.go:414] "containerMap: removing container" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492020 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492079 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492087 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492092 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c032c362-3944-42cf-8576-2c74c8f9dca2" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492099 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492105 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492111 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="66834ead-f417-4228-bd5f-a5cde6ca346f" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492117 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b23fab-1eba-43bf-ae70-bff4daba1fcc" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492121 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d5ff2f-96f2-4477-b319-219840d3db21" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492126 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5f059b-0888-4996-9ab2-45bf7606cd44" containerName="payload-pre-processing" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492131 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492136 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492141 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ab5024-b7a9-496b-951d-ea388be139c5" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492148 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="6309d2d6-8637-4a0f-9520-7024b8957e5e" containerName="main" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492152 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="9194f4ba-5aea-450b-8ac3-efefa4b1e743" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492158 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492164 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d4fd17-1756-47a6-81df-e3d7174555f0" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492169 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77b4f03-6347-4591-919d-0aa3ba2453fc" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492174 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbacae8f-ac18-44ba-aedb-6ecb049209dc" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492180 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="b639357e-4f63-439a-b1b6-2e32bfa296a2" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492186 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="92343dce-b0c6-44f8-a913-e718e6eb0ad7" containerName="maas-api" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492246 4991 cpu_manager.go:414] "containerMap: removing container" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492251 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39de1aba-6701-42bf-bcfd-d2327f97f5a5" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492264 4991 cpu_manager.go:414] "containerMap: removing container" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492269 4991 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492350 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4cc59e-42c4-4e5b-b996-2b4cf886879b" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492357 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="843f57c2-8883-4f2a-aec5-3e8e940ad5fd" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492362 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae9d702-f807-4040-b5e8-e140bead935f" containerName="istio-proxy" Oct 30 22:58:45.492809 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.492367 4991 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a6c4db-a0b8-442c-90f4-52f3cfe31a29" containerName="istio-proxy" Oct 30 22:58:45.495505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.495486 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:45.550782 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.550758 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mpzc/must-gather-8nz7d"] Oct 30 22:58:45.598840 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.598816 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5mpzc\"/\"default-dockercfg-w42bs\"" Oct 30 22:58:45.598959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.598825 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mpzc\"/\"openshift-service-ca.crt\"" Oct 30 22:58:45.598959 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.598860 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mpzc\"/\"kube-root-ca.crt\"" Oct 30 22:58:45.601109 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.601091 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7d8m9/must-gather-5hm9d"] Oct 30 22:58:45.604625 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.604611 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:45.654291 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.654266 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/must-gather-5hm9d"] Oct 30 22:58:45.667457 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.667432 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bb815d3-d799-429f-8ab0-13b96f595c06-must-gather-output\") pod \"must-gather-8nz7d\" (UID: \"8bb815d3-d799-429f-8ab0-13b96f595c06\") " pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:45.667606 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.667586 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrrbx\" (UniqueName: \"kubernetes.io/projected/8bb815d3-d799-429f-8ab0-13b96f595c06-kube-api-access-vrrbx\") pod \"must-gather-8nz7d\" (UID: \"8bb815d3-d799-429f-8ab0-13b96f595c06\") " pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:45.694514 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.694497 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7d8m9\"/\"openshift-service-ca.crt\"" Oct 30 22:58:45.694720 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.694703 4991 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7d8m9\"/\"kube-root-ca.crt\"" Oct 30 22:58:45.755821 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.755768 4991 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7d8m9\"/\"default-dockercfg-qb7pg\"" Oct 30 22:58:45.768088 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.768057 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrrbx\" (UniqueName: \"kubernetes.io/projected/8bb815d3-d799-429f-8ab0-13b96f595c06-kube-api-access-vrrbx\") pod \"must-gather-8nz7d\" (UID: \"8bb815d3-d799-429f-8ab0-13b96f595c06\") " pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:45.768192 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.768153 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bb815d3-d799-429f-8ab0-13b96f595c06-must-gather-output\") pod \"must-gather-8nz7d\" (UID: \"8bb815d3-d799-429f-8ab0-13b96f595c06\") " pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:45.768259 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.768238 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmt7\" (UniqueName: \"kubernetes.io/projected/7d6d38f6-836d-435e-9d76-6797274e2578-kube-api-access-hlmt7\") pod \"must-gather-5hm9d\" (UID: \"7d6d38f6-836d-435e-9d76-6797274e2578\") " pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:45.768341 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.768286 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7d6d38f6-836d-435e-9d76-6797274e2578-must-gather-output\") pod \"must-gather-5hm9d\" (UID: \"7d6d38f6-836d-435e-9d76-6797274e2578\") " pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:45.768509 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.768489 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bb815d3-d799-429f-8ab0-13b96f595c06-must-gather-output\") pod \"must-gather-8nz7d\" (UID: \"8bb815d3-d799-429f-8ab0-13b96f595c06\") " pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:45.831145 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.831118 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrrbx\" (UniqueName: \"kubernetes.io/projected/8bb815d3-d799-429f-8ab0-13b96f595c06-kube-api-access-vrrbx\") pod \"must-gather-8nz7d\" (UID: \"8bb815d3-d799-429f-8ab0-13b96f595c06\") " pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:45.869295 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.869263 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7d6d38f6-836d-435e-9d76-6797274e2578-must-gather-output\") pod \"must-gather-5hm9d\" (UID: \"7d6d38f6-836d-435e-9d76-6797274e2578\") " pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:45.869463 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.869447 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmt7\" (UniqueName: \"kubernetes.io/projected/7d6d38f6-836d-435e-9d76-6797274e2578-kube-api-access-hlmt7\") pod \"must-gather-5hm9d\" (UID: \"7d6d38f6-836d-435e-9d76-6797274e2578\") " pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:45.869589 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.869571 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7d6d38f6-836d-435e-9d76-6797274e2578-must-gather-output\") pod \"must-gather-5hm9d\" (UID: \"7d6d38f6-836d-435e-9d76-6797274e2578\") " pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:45.931296 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:45.931268 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmt7\" (UniqueName: \"kubernetes.io/projected/7d6d38f6-836d-435e-9d76-6797274e2578-kube-api-access-hlmt7\") pod \"must-gather-5hm9d\" (UID: \"7d6d38f6-836d-435e-9d76-6797274e2578\") " pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:46.109916 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:46.109857 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mpzc/must-gather-8nz7d" Oct 30 22:58:46.218067 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:46.218035 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/must-gather-5hm9d" Oct 30 22:58:46.492858 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:46.492835 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mpzc/must-gather-8nz7d"] Oct 30 22:58:46.493441 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:58:46.493414 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bb815d3_d799_429f_8ab0_13b96f595c06.slice/crio-ca627edeb41f3cdbe624f1b54c4fdb4a13085255a857c926b13e324f1552e506 WatchSource:0}: Error finding container ca627edeb41f3cdbe624f1b54c4fdb4a13085255a857c926b13e324f1552e506: Status 404 returned error can't find the container with id ca627edeb41f3cdbe624f1b54c4fdb4a13085255a857c926b13e324f1552e506 Oct 30 22:58:46.495020 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:46.495001 4991 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:58:46.591212 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:46.591192 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/must-gather-5hm9d"] Oct 30 22:58:46.593516 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:58:46.593496 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d6d38f6_836d_435e_9d76_6797274e2578.slice/crio-d303c174ab257362b8766417335fe8e406da715f7465e00afab95753a3951f1e WatchSource:0}: Error finding container d303c174ab257362b8766417335fe8e406da715f7465e00afab95753a3951f1e: Status 404 returned error can't find the container with id d303c174ab257362b8766417335fe8e406da715f7465e00afab95753a3951f1e Oct 30 22:58:47.491101 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:47.491061 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mpzc/must-gather-8nz7d" event={"ID":"8bb815d3-d799-429f-8ab0-13b96f595c06","Type":"ContainerStarted","Data":"ca627edeb41f3cdbe624f1b54c4fdb4a13085255a857c926b13e324f1552e506"} Oct 30 22:58:47.492301 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:47.492278 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/must-gather-5hm9d" event={"ID":"7d6d38f6-836d-435e-9d76-6797274e2578","Type":"ContainerStarted","Data":"d303c174ab257362b8766417335fe8e406da715f7465e00afab95753a3951f1e"} Oct 30 22:58:47.913942 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:58:47.913859 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:58:48.501014 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:48.500976 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/must-gather-5hm9d" event={"ID":"7d6d38f6-836d-435e-9d76-6797274e2578","Type":"ContainerStarted","Data":"5d31b84adc6a0d804f405f516e778f771742dc959844e8b4540ecfca4f243046"} Oct 30 22:58:48.501189 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:48.501038 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/must-gather-5hm9d" event={"ID":"7d6d38f6-836d-435e-9d76-6797274e2578","Type":"ContainerStarted","Data":"bfb98e3e2337ec989dffae13465a93777b4fcefaa6995e0f3edf3330722d50c4"} Oct 30 22:58:48.620222 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:48.620149 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7d8m9/must-gather-5hm9d" podStartSLOduration=2.201951869 podStartE2EDuration="3.620129822s" totalImagesPullingTime="1.418177953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:48.618236597 +0000 UTC m=+4348.917646477" watchObservedRunningTime="2026-10-30 22:58:48.620129822 +0000 UTC m=+4348.919539705" Oct 30 22:58:53.541139 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:53.541111 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mpzc/must-gather-8nz7d" event={"ID":"8bb815d3-d799-429f-8ab0-13b96f595c06","Type":"ContainerStarted","Data":"480233394c630356ed2a77e9206f693324c46fdf6e91a18fdd2e02871484d194"} Oct 30 22:58:53.541550 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:53.541152 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mpzc/must-gather-8nz7d" event={"ID":"8bb815d3-d799-429f-8ab0-13b96f595c06","Type":"ContainerStarted","Data":"eead11667923649f77deb49ae954f577e85b889ffde48f99d7ec0bc3dd199746"} Oct 30 22:58:53.657915 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:53.657861 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5mpzc/must-gather-8nz7d" podStartSLOduration=1.990178392 podStartE2EDuration="8.657849062s" totalImagesPullingTime="6.66767067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:53.656163801 +0000 UTC m=+4353.955573681" watchObservedRunningTime="2026-10-30 22:58:53.657849062 +0000 UTC m=+4353.957258939" Oct 30 22:58:55.289466 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:55.289437 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4zhvv_d3280f18-2d95-4303-8f34-5a2fe91671e6/global-pull-secret-syncer/0.log" Oct 30 22:58:57.107856 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:57.107828 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2jxfz_6ffd4edc-ce40-4a48-b398-c2d907a7cd0a/connection-checker/0.log" Oct 30 22:58:57.543561 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:57.543524 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dll8d_177b7d0a-d868-497a-b648-23228667c743/connection-checker/0.log" Oct 30 22:58:58.012864 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:58.012831 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-p2dr2_0abc3e92-4ede-4421-ab8d-91477bd6b85e/connection-checker/0.log" Oct 30 22:58:58.738382 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:58.738068 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:58:58.853887 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:58.853843 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:58:59.425714 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:59.409373 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r2dj5_fad6427c-0014-4c83-9ff9-ce1d9ba0aa55/konnectivity-agent/0.log" Oct 30 22:58:59.859356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:59.605108 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:58:59.859356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:58:59.747916 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:59:00.196939 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:00.196861 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-17.us-west-2.compute.internal_44d4a0aec485b9ad44110c51253a50d0/haproxy/0.log" Oct 30 22:59:04.124721 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:04.124669 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:59:04.246418 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:04.246385 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:59:05.692832 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:05.692796 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57dbf74b78-dx9mj_c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd/authorino/0.log" Oct 30 22:59:06.029188 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:06.029142 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdbtw_98b55f48-d625-4a7f-bbd7-98e462d435b5/manager/0.log" Oct 30 22:59:06.132041 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:06.131936 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:59:06.253389 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:06.253360 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:59:06.435132 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:06.435055 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c8sn_0ec987ff-0828-4a26-8926-874d8a47cbc7/manager/0.log" Oct 30 22:59:07.450078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.450037 4991 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz"] Oct 30 22:59:07.460068 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.460041 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.508414 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.508379 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz"] Oct 30 22:59:07.628044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.628004 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-podres\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.628044 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.628047 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-proc\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.628300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.628068 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-lib-modules\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.628300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.628203 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scs8k\" (UniqueName: \"kubernetes.io/projected/96c14257-b794-4e83-ac36-67a5cc877e5b-kube-api-access-scs8k\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.628300 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.628262 4991 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-sys\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.728933 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.728880 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-podres\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.728944 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-proc\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.728976 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-lib-modules\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.729030 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scs8k\" (UniqueName: \"kubernetes.io/projected/96c14257-b794-4e83-ac36-67a5cc877e5b-kube-api-access-scs8k\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729136 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.729077 4991 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-sys\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.729211 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-sys\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.729271 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-podres\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.729314 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-proc\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.729427 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.729387 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96c14257-b794-4e83-ac36-67a5cc877e5b-lib-modules\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.793921 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.793891 4991 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scs8k\" (UniqueName: \"kubernetes.io/projected/96c14257-b794-4e83-ac36-67a5cc877e5b-kube-api-access-scs8k\") pod \"perf-node-gather-daemonset-58nrz\" (UID: \"96c14257-b794-4e83-ac36-67a5cc877e5b\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:07.940537 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:07.940496 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhmj8_0bf5a041-29ee-46b9-88ae-a1bfd29d24c2/manager/0.log" Oct 30 22:59:08.090505 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:08.090429 4991 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:08.469788 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:08.469763 4991 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz"] Oct 30 22:59:08.471956 ip-10-0-137-17 kubenswrapper[4991]: W1030 22:59:08.471925 4991 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod96c14257_b794_4e83_ac36_67a5cc877e5b.slice/crio-9ea375eff11198fb3f7f4f2e897e99e0147c2ceb7dce7c231105dc3bdc42c298 WatchSource:0}: Error finding container 9ea375eff11198fb3f7f4f2e897e99e0147c2ceb7dce7c231105dc3bdc42c298: Status 404 returned error can't find the container with id 9ea375eff11198fb3f7f4f2e897e99e0147c2ceb7dce7c231105dc3bdc42c298 Oct 30 22:59:08.750494 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:08.750455 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" event={"ID":"96c14257-b794-4e83-ac36-67a5cc877e5b","Type":"ContainerStarted","Data":"55bfb6e73e57dfa2622e5b175877eb9f9cebe0e4b863732248ffb2e0262be8cd"} Oct 30 22:59:08.750690 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:08.750550 4991 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" event={"ID":"96c14257-b794-4e83-ac36-67a5cc877e5b","Type":"ContainerStarted","Data":"9ea375eff11198fb3f7f4f2e897e99e0147c2ceb7dce7c231105dc3bdc42c298"} Oct 30 22:59:08.750690 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:08.750580 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:08.868613 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:08.868505 4991 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" podStartSLOduration=1.8684902920000002 podStartE2EDuration="1.868490292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:08.868037535 +0000 UTC m=+4369.167447415" watchObservedRunningTime="2026-10-30 22:59:08.868490292 +0000 UTC m=+4369.167900173" Oct 30 22:59:09.810421 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:09.810384 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:59:09.931726 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:09.931699 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:59:11.691433 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:11.691401 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569467969-472sq_731fd329-f519-4400-af1b-dfdf91e13693/manager/0.log" Oct 30 22:59:11.807610 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:11.807576 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9zdg4_a108075d-1569-471d-9f71-287c7d156af2/server/0.log" Oct 30 22:59:14.314591 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:14.312623 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-dvh4d_67d09fe6-fb0a-43cc-8b20-98cdb22eea2a/cluster-monitoring-operator/0.log" Oct 30 22:59:14.470100 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:14.470068 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz_e0038133-9d86-4cf5-a75f-539470632d85/istio-proxy/0.log" Oct 30 22:59:14.771141 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:14.771115 4991 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-58nrz" Oct 30 22:59:16.138078 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:16.138038 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph_ec12978d-ab70-452f-8b7b-4954fa7fce4c/istio-proxy/0.log" Oct 30 22:59:16.307783 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:16.307744 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz_e0038133-9d86-4cf5-a75f-539470632d85/istio-proxy/0.log" Oct 30 22:59:16.750974 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:16.750941 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8c9db44f-llgx4_a547a8cd-8909-475e-a03d-15b409877777/router/0.log" Oct 30 22:59:16.755356 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:16.755306 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f47h_1d16bd55-bd8f-42df-9bac-f93265002009/node-exporter/0.log" Oct 30 22:59:17.161473 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:17.161353 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f47h_1d16bd55-bd8f-42df-9bac-f93265002009/kube-rbac-proxy/0.log" Oct 30 22:59:17.568792 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:17.568755 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f47h_1d16bd55-bd8f-42df-9bac-f93265002009/init-textfile/0.log" Oct 30 22:59:17.917077 ip-10-0-137-17 kubenswrapper[4991]: E1030 22:59:17.916983 4991 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhs7l" containerName="extractor" Oct 30 22:59:17.997730 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:17.997704 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5579cb49c9-wjjph_ec12978d-ab70-452f-8b7b-4954fa7fce4c/istio-proxy/0.log" Oct 30 22:59:18.637397 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:18.637112 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8c9db44f-llgx4_a547a8cd-8909-475e-a03d-15b409877777/router/0.log" Oct 30 22:59:22.345446 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:22.345409 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57dbf74b78-dx9mj_c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd/authorino/0.log" Oct 30 22:59:22.465976 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:22.465944 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdbtw_98b55f48-d625-4a7f-bbd7-98e462d435b5/manager/0.log" Oct 30 22:59:22.662073 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:22.661983 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c8sn_0ec987ff-0828-4a26-8926-874d8a47cbc7/manager/0.log" Oct 30 22:59:23.714475 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:23.714420 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhmj8_0bf5a041-29ee-46b9-88ae-a1bfd29d24c2/manager/0.log" Oct 30 22:59:24.950689 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:24.950650 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57dbf74b78-dx9mj_c6e9aa35-1c5f-4ff2-a5b7-eac59cf341bd/authorino/0.log" Oct 30 22:59:25.069425 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:25.069389 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdbtw_98b55f48-d625-4a7f-bbd7-98e462d435b5/manager/0.log" Oct 30 22:59:25.277923 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:25.277895 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c8sn_0ec987ff-0828-4a26-8926-874d8a47cbc7/manager/0.log" Oct 30 22:59:26.370806 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:26.370763 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rhmj8_0bf5a041-29ee-46b9-88ae-a1bfd29d24c2/manager/0.log" Oct 30 22:59:31.162619 ip-10-0-137-17 kubenswrapper[4991]: I1030 22:59:31.162581 4991 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c57xlwjz_e0038133-9d86-4cf5-a75f-539470632d85/istio-proxy/0.log"